# Documentation Guides and tutorials to get you started with Taskforce.sh ## Introduction Taskforce.sh is a front end for managing Bull/BullMQ instances. It provides tools to efficiently manage and debug your queues as well as a hosting service for redis instances. The tools provided are useful during development, so that jobs can be verified for proper execution. They are also useful in production, where it is important to keep an eye on how jobs are being processed. Sometimes jobs fail due to transient errors or bugs in the code. In those cases, the jobs can be analyzed and retried once the underlying problem is resolved. For documentation on BullMQ you can visit and Bull is located [here](https://github.com/OptimalBits/bull#documentation). # Getting started There are several ways to get your queues manageable under Taskforce.sh, depending on how you are hosting your Redis instances and/or your security requirements. Connecting your Redis instances to Taskforce.sh is quite easy, you can connect your existing Redis databases to Taskforce.sh using a direct connection to your Redis host or by using the provided "[taskforce connector](https://docs.taskforce.sh/getting-started/taskforce-connector)". In the following pages we will go through the different alternatives. # Direct Redis Connection ## Direct Redis connection Connecting to an existing Redis instance that is running in a publicly opened server is straightforward, just create a new connection in the left side panel clicking on the + symbol: ![](/files/-MGsXTXNAQSfWViEIwcl) {% hint style="danger" %} It is crucial that your existing Redis instance is configured with the setting **maxmemory: noeviction** for Bull/BullMQ to work without issues. {% endhint %} You can now fill the connection details. We **highly** recommend you to use TLS for encrypted communication with your database, please refer to your provider on how to activate it if it is not activated already, normally the TLS enabled connection will be available in a different port than the standard unencrypted connection: ![Dialog for configuring a connection to a redis server](/files/-MGsY8ecSumlcaEQReNo) {% hint style="info" %} The Redis passwords are always stored encrypted using highly secure cryptographic standards. {% endhint %} #### Securing by IP If you have a Redis connection hosted somewhere, one important security practice is to limit which IP**'**s are allowed to connect to your instance. Taskforce.sh will connect to your instances using the IP: **18.193.189.120** So you will need to configure your Redis service to allow the IP above so that Taskforce.sh can access the queues on your instance. # Taskforce connector ## Taskforce connector The taskforce connector is a small command line tool that is used for connecting securely a redis instance to taskforce.sh. The connector opens a websocket connection (TLS encrypted) on some instance close to your redis connection, can be inside a secured VPC without any public access, as long as the instance running itself has access to the internet. The connector is particularly useful if you want to connect redis instances running on your localhost. ### Testing the connector You can easily test the connector locally before using it in a more sophisticated production environment. You will need a running local redis instance for the following instructions to work. Start by installing the connector globally: ```bash $ yarn global add taskforce-connector ``` In order to use the connector you will need an API token, you can find the one belonging to your account in the Account view in taskforce.sh: ![](/files/-MGt3xFDEeQBgW50cvNN) Using this token you can now call the connector to establish the connection to taskforce.sh: ```bash $ taskforce -n "My Redis" -t 40bf2f58-b684-484b-9379-581f710e9877 ``` The connection will appear automatically in the left sidebar, and you will have access to all the queues in that redis instance: ![](/files/-MGt7IYAhuSRbhjq0fG_) The connector has several other settings, for instance you can specify connection details for your redis instance: ```bash $ taskforce -n "My Redis" -h redis.host -p 6379 -passwd xxxx ``` {% hint style="info" %} you need to have enough connections quota in taskforce.sh for the connector to work. {% endhint %} ### Using a docker container We also provide a minimal docker container that can be conveniently deployed in your infrastructure. All required settings are exposed as environment variables, see this `docker-compose.yaml` file as an example on how to run the connector and a redis instance connected to it: ```yaml version: "3.7" services: redis: image: "redis:alpine" ports: - 7000:6379 taskforce-connector: build: https://github.com/taskforcesh/taskforce-connector.git#master environment: TASKFORCE_CONNECTION: "local taskforce provisioner" TASKFORCE_TEAM: "Taskforce" TASKFORCE_TOKEN: ${TASKFORCE_TOKEN} REDIS_HOST: redis ``` ### Teams The taskforce connector also allows you to specify a team, as in the yaml above or using the command line. For the following to work you need to create a Team in taskforce.sh first. ```bash $ taskforce --team "my awesome team" -n "My Redis" -t 2cfe6a1b-5f0e-466f-99ad-12f51bea79a7 ``` {% hint style="info" %} You can find the latest up-to-date instructions on how to use the connector here: {% endhint %} ### Queues Discovery By default, the connector will issue a scan to find the queues that are present in the given Redis instance. This scan filters the keys by key type (only string keys are considered), and that match a given pattern (`prefix:queue-name:id`). This method is automatic and works most of the time, completely transparent to the user. Depending on the Redis instance, or if the amount of keys in the instance is very large, the time to do the queue discovery can be too long. There is a maximum timeout of 40 seconds, after that, the connector will not continue trying to find more queues in the instance. #### Specifying queue names manually It is possible to manually specify the queues that we want to give access to Taskforce.sh. This is useful in order to avoid a possible slow queue discovery, or to protect queues that should not be accessible by all the members of a given team. Notice that you could have different connectors for the same Redis instance for different teams. You can specify the queues either in the command line or in a separate text file. In the command line separate every queue name with commas (do not add any spaces between queue names): ``` $ taskforce -t 2cfe6a1b-5f0e-466f-99ad-12f51bea79a7 --queues monitoring,payments ``` If your queues are not using the standard queue prefix "bull", you can specify a different prefix as well: ``` $ taskforce -t 2cfe6a1b-5f0e-466f-99ad-12f51bea79a7 --queues foobar:monitoring,payments ``` If your queue names include non-ASCII characters or spaces you can wrap the list in quotes: ``` $ taskforce -t 2cfe6a1b-5f0e-466f-99ad-12f51bea79a7 --queues "my queue" ``` If you prefer to specify the queue names in a separate file you can just add the queue names separated by newlines: {% code title="// my-queues.txt" %} ``` foobar:monitoring payments etc ``` {% endcode %} And specifying the filename with `--queuesFile` : ``` $ taskforce -t 2cfe6a1b-5f0e-466f-99ad-12f51bea79a7 --queuesFile my-queues.txt ``` # Account In this section we will cover different aspects that you can configure in your account. The account page is divided into different tabs, and we will cover each one here. # Settings This is the settings tabs where you can configure some account specific options. #### Multi-Factor Authentication If you want to provide the maximum security to your account, specially if you are using a user/password based account, you should enable MFA. MFA will require a separate hardware device when the system detects that the risk is higher than normal, for example if you are trying to log in for a different location that you usually do. Start by clicking on the "Enroll in MFA" button.
You will be asked if you want to enable it for sure. Note that once enable it cannot be easily disabled, contact support if you need to do it.
You will be prompted to log-in again, but this time you will be able to configure MFA.
For the next step you are going to need your Authenticator App in any mobile device:
After inserting the OTP code you will be all set, and next time you login you will need to have your authenticator app.
{% hint style="danger" %} It is highly recommended to remember the device for 30 days if you are login in from a trusted PC or Mac. {% endhint %} # Connections A connection in taskforce represents a connection to a redis instance. Therefore a connection can include none or several queues. The connection panel in the sidebar allows you to view all the queues available in a given redis database and to choose any queue for performing more advanced operations. ![](/files/-MWP7Y9IWpZFhIekBZGx) Connections can be purchased separately if using external redis hosts, or they are included if you provision your redis instances in Taskforce.sh. Once you have a working connection it will have a green status icon indicating that the connection has successfully connected to redis. Initially the connection will fetch the queues available in the redis database, however it is necessary to refresh manually if new queues are added or removed. Sometimes the connection status icon may light red. In this case the connection has lost connectivity with the redis database. We try to give some hint on why this may have happened. If you are unable to sort out the problem, please reach support for assistance. ![](/files/-MWP7jzwI4JwsuNDRP1J) # Queues When selecting a queue from the connection's panel, the main queue view will open up. Here you can get some overview of your queue, including metrics for the last processed jobs, as well as tables that shows the jobs in their different status. ![](/files/-MWP7t9we6fjmJ5Hq4fh) From this view it is possible to perform some queue-wide operations such as: adding new jobs, global **pause/resume** and queue **deletion**. Keep in mind that deletion is a destructive operation that cannot be undone. ![](/files/-MWP8oDLQETRe13ro8Xp) The tables are not auto-updated so you will need to hit the refresh button in order to get the latest view of a given queue. ### Adding jobs It is possible to add new jobs to a given queue. This is useful when trying out the workers in different contexts. In order to add a new job just press on the + icon in the queue bar: ![](/files/-MeOnzdmP4_RowonfqKb) {% hint style="info" %} You need at least version 1.15.0 of the Taskforce connector or a direct Redis connection for the Add job feature to be available. {% endhint %} Clicking on the icon will open the "Add job" dialog: ![](/files/-MeOoIueROWTcEpvHKAF) You will be able to name your job and optionally change the delay or max. attempts settings. For the data portion of the job you get a nice JSON editor. Just fill in your data and click on "Add": ![](/files/-MeOpHjTbHg1CquWu0Ik) The job will be added directly to the queue and if the queue has running workers it will be processed accordingly. {% hint style="info" %} Leaving the job name empty will produce a job with the special name \_\_*default\_\_* which in Bull v3 is the default name when not providing one. {% endhint %} # Jobs The jobs view allows you to examine jobs that are in the queue or that have been processed by the workers. You can select any of the possible states where a job can be in a queue to discover which jobs are in said states: ![](/files/-MWPC2Mylj0tiaOoPqsj) Jobs can be opened in order to get useful information such as: options, data, result values, logs, error callstacks and reasons for failing: ![](/files/-MWPCYYu1abAioQeAvrH) From this view it is also possible to perform operations on the individual jobs, or on a certain selection of the jobs. ![Job action menu](/files/-MWPDQTg-8zUug-mHJre) For example, a common action is to retry failed jobs after resolving the underlaying reason for the failure. #### Updating jobs It is possible to edit the data of existing jobs if they are in *compatible* states such as "Waiting", "Delayed" or "Failed". You update a job by clicking on the "Edit" icon: ![](/files/-MePCBLYLKkaVBlJGyw7) You will get up an update job dialog: ![](/files/-MeOue3sr7sE8FZD4U2v) Just perform any change you want on the data field as JSON and click on Update: ![](/files/-MeOvLaQPhtaGSK6Rcrn) # Workers The workers view gives you a list of all the actual workers connected to a given queue. This is useful for verifying that the intended number of workers are really connected and processing jobs. If the list is missing some expected workers, it could be because they are offline or have lost their connection to the Redis database. ![](/files/-MWPE9CaQ3vNCi4u0Vyz) {% hint style="info" %} The workers represent actual workers instances connected to Redis, independently on the concurrency setting used in the workers. {% endhint %} {% hint style="danger" %} Currently, due to an underlying issue in Bull/BullMQ, the workers table shows incorrect results if you are reusing connections. {% endhint %} # Metrics Taskforce.sh provides some useful metrics for your queues. The metrics view provides you a glimpse on how your queues are performing, this will allow you to quickly discover if some queue is running slower than what you expect, so that you can take actions to resolve the problems. The first section of the metrics view shows the global statistics for the queue, the total number of jobs "In Queue", that is, jobs that are currently either waiting for being processed or being processed right now. ![](/files/FdQTIiMFOuFh7oKAW3QD) It also shows you how many jobs in total have been completed and failed, note that for this metric to be accurate you must either enable metrics on your queues or keep all the completed and failed jobs (usually not recommended since it will consume more and more space from your Redis instance). The second second section shows real-time metrics for up to the last 2 weeks with a minimum granularity of 1 minute, in other words, you can see how many jobs per minute your queue is completing (or failing). ![](/files/2vexisBKvYq1qRKwAdOk) The third section displays performance statistics. Currently these statistics requires you to keep at least 1k completed jobs on the queue. ![](/files/cQi31pBcFuH1tiIPkmMb) ### Enabling Queue Metrics In order to enable metrics collection on the queues you need to add the metrics options to your queue. #### Bull In Bull you should enable the metrics on every queue instance that defines a processor: ```typescript const Queue = require('bull'); new Queue("queue-with-metrics", { metrics: { maxDataPoints: Queue.utils.MetricsTime.TWO_WEEKS } }) queue.process('myprocessor'); ``` {% hint style="info" %} We recommend to keep two weeks of metrics, which would require around 120Kb of RAM from your Redis instance. {% endhint %} #### BullMQ In BullMQ you should enable the metrics on every worker: ```typescript import { Worker, MetricsTime } from 'bullmq' const worker = new Worker('queue-with-metrics', { connection, metrics: { maxDataPoints: MetricsTime.TWO_WEEKS, }, }); ``` # Monitoring Taskforce.sh provides a powerful monitoring facility. All connections can be monitored for different types of common issues that can happen in the queues that are part of the Redis instance associated with a connection. Monitors that are activated on your connections generate alerts if the monitor detects an issue in any of the queues in the connection. Alerts are visible from the dashboard and include information regarding the cause of why the alert has been triggered. All triggered alerts send also an email to the owner of the connection. Currently, we support the following monitor types: * Connection monitor * Failed jobs monitor * Missing workers monitor * Max Memory monitor * Backlog monitor Over time we will add more monitor types for other common errors, you are welcome to suggest new monitor types here: {% hint style="info" %} Note that by **default** the monitors are **disabled** and must be enabled manually on every connection. {% endhint %} You can access a connection's monitors by clicking on the connection name:
This will open the connection dialog and you will be able to choose which monitors to activate: ![](/files/0IoFs1T5jVLATE3ESzb9) # Enable notifications In order to get a notification every time a monitor triggers an alert, we need to enable which notification backends to use in every connection. You will find a Notifications section at the end in the notifications tab:

Notifications setup

Currently, when enabling the Email notification, the email of the **owner of the connection** will be used as the destination address. # Slack How to configure notifications for slack You can easily use a Slack channel as the target for your alerts. For this, you will need to create an Incoming Hook URL that can be used for delivering messages to a specific Slack channel. Start by going to [api.slack.com](https://api.slack.com) and select an existing App or create a specific App to be used for Taskforce.sh:
Now, open your application and go to the Basic Information section and click the "Incoming Webhooks" card.
From the Incoming Webhooks you can just create a new hook for a given channel:
You can then select which channel to post the alerts on:
Your hook URL will now appear on the list of Webhook URLs
You can just copy the given URL and paste it into your monitor's settings:
That's all, from now on, every time an alert is produced you will get a message on the given slack channel:
# Pagerduty The notification system also supports sending alerts to PagerDuty. When an alert is triggered, Taskforce.sh calls PagerDuty's Events API, and the alert becomes available in your PagerDuty dashboard. Normally, all members of the team also receive an alert email.

Pagerduty configuration

{% hint style="info" %} The alerts produced by the monitoring system are also kept in the Taskforce.sh alerts tab. To acknowledge them, you need to do so manually, or no new alerts will be produced during the next 24 hours for a given monitor in a given queue. {% endhint %} # Connection monitor The connection monitor checks that the access to your Redis instance works properly. If for some reason the Redis host is not reachable, either because the instance is down or there is a connectivity error between Redis and the Taskforce.sh servers, then an alert will be triggered. This is the most critical monitor since it implies that the Redis server may be down and an action to bring it back online may be needed. When a connection alert is in "Triggered" status, no more alerts will be triggered until the alert is acknowledged or a full day has passed. Furthermore, if the connection is not reachable, none of the other monitors will work either so it is really important to make sure the connection is back online as soon as this alert is triggered. ![](/files/9HD3yakZVoj8z38FGbuL) # Failed jobs monitor This monitor checks if there are new jobs that have failed. The monitor will trigger an alert as soon as it detects that there is a job at the top of the failed jobs that have not triggered an alarm before. {% hint style="danger" %} For this monitor to work properly you need to keep at least 1 failed job in your queues with the removeOnFailed option set to false, which is the default, or to a number equal or larger than 1. {% endhint %} Since it is not uncommon that when one job fails several others fail too, the monitor will not trigger new alerts until the existing triggered alert is acknowledged. {% hint style="info" %} The alerts are triggered per queue, so you can have many alerts in the "triggered" status belonging to different queues. {% endhint %} ![](/files/oZ7GOHc5bmf4Yan4YZC9) # Missing workers monitor The missing workers monitor checks if there are enough workers running for your queues. For example, you may have designed your infrastructure to have at least 4 workers running at any given time, for redundancy and performance, you could then enable this monitor and configure it to trigger alerts if the number of workers goes below 2. As with other monitors, this monitor will not trigger new alerts for queues that have alerts in triggered status. ![](/files/uBG1NZfllfAkE67B42Nk) # Max Memory monitor The Max Memory monitor checks if the amount or memory used in your Redis instance has grown over a certain limit. This monitor is quite important to activate on production systems as it will alert in time for upgrading or cleaning the Redis instance before it becomes full and it starts malfunctioning.
The recommended threshold should be something around 80% of its full capacity. As all Redis hosts do not always report the maximum memory capacity, this monitor requires you to specify a certain amount of Megabytes that would be your threshold. Remember that 1Gb is about 1024Mb. # Backlog Monitor The Backlog monitor checks if the number of jobs waiting to be processed has exceeded a given maximum. As jobs in the backlog we include "waiting jobs", "prioritized jobs" and "delayed jobs".
# Alerts Alerts are produced by monitors and can be found on the "Alerts" tab in your connection dialog. ![](/files/wqkmsbz3eJYXNfTiJiRt) When alerts are triggered, the owner of the connection will also receive an email with the notification that a certain monitor has triggered the alert. ![](/files/nSwlkvu8jyE6MdHhguXO) Most alerts require manual acknowledgement before a given monitor type will produce new alerts for the same queue, so it is advised to investigate as soon as possible why the alert was triggered and correct any issues before the alert is acknowledged. # Organizations Organizations is a feature in Taskforce.sh that allows sharing connections between members of your teams. You can only create an organization if you have a valid subscription. Start by going to the "Teams" page:
Continue by writing the name of your organization. Note that names must be unique within Taskforce.sh:
Click Save to create the organization. You will now be able to add any number of teams to the given organization:
### Teams You can proceed by creating any number of teams. The teams will allow you to add members to your organization based on roles. It is also useful to organize connections in different types or purpose, for example you can have a team for production connections and another for development connections.
Members are added to the teams by sending them an invitation. Note that the user to be invited must have already signed in to Taskforce.sh (no subscription is needed though). Click on the "Invite Member" button to add new members.
### Roles There are currently 5 roles to choose from. * Guest - This role only allows read rights. Meaning that a guest can see all the queues, and their contents, but is not allowed to change anything. * Operator - An operator has some minimal write rights, such as being able to "retry" and "promote" jobs, and nothing else. * Regular - Members with this role are able to view and interact with all the connections that are part of the team. * Manager - These members are also able to add, edit and remove connections, including configuring Monitors and Alerts. * Admin - Admins have the same rights as Manager and Regular members but can also invite new team members as well as remove them and change other member's roles.
### Invitations An invited user will receive an email with instructions and will also appear in the invitations tab:
When the invited user goes to the Teams page, an invitation message will appear:
### Adding Connections Back in the dashboard, all the organization members will see the organizations and teams that they belong to. They will also be able to add connections to any team if they have the proper rights (either as a member of a team with the Admin or Manager role, or as the owner of the organization the team he/she belongs to).
Clicking on the + button will open the connection dialog.
After saving it it will appear under the team:
### Using the Taskforce Connector It is also possible to add connections to a team using the taskforce-connector by specifying the team name and using the user's token on Account->Connection Token. For example: ```sh $ taskforce -t 32852395-5e50-44c2-81e5-e6aaa73fc235 --team Development ``` By using the connector, an Organization's administrator can create several teams, maybe one team per user so that they can connect to their local queues, which is very useful under development. # Custom roles Define custom roles for your team members It is also possible to define custom roles if the predefined ones do not fulfill your requirements. In the right tab on the organization view, you will see a tab "Custom Roles".
In this tab you can see all the custom roles that you have created. You can create as many as you need, just hit the create button to get a new dialog that will allow you to choose any permissions you want for your new role. Note that custom roles are based on a built-in role, this will help you to faster create new roles, for instance if there is a built-in role that almost fullfill your needs but you need to customize it a bit. You can either inherit the permission from the base role, enable it or disable it. This will give you quite good granularity on the actions that are allowed for any given role.
When you have a created a role, you will now be able to use it when inviting new members or change existing members to use the new role.
The user with the new custom role attached will now be limited or allowed to access operations enabled by this new role. # BullMQ Pro BullMQ Pro is the commercial version of BullMQ with a growing set of advanced features and support from the authors of the original BullMQ library. In order to gain access to this version you need to subscribe to the BullMQ Pro license in your Taskforce.sh subscription panel. After subscribing you will now be able to generate Npm tokens that you can use to install BullMQ Pro on your applications. ![](/files/pBANrnDoKDbiZWmyuLJz) {% hint style="danger" %} Do not commit these tokens into your code repositories, instead always use env variables for security. {% endhint %} Please note that every subscription is per organization, and you are not allowed to share these tokens to other organizations than the one paying for it. For information on how to use BullMQ Pro please refer to the official [documentation](https://docs.bullmq.io/bullmq-pro). If you need support please reach us at the support [site](https://github.com/taskforcesh/bullmq-pro-support). # Connector Pro It is possible to bring connections that use BullMQ Pro to the dashboard by using the Pro version of the connector. Currently, the connector does not have a CLI version but you can easily use it programmatically. First, you need to install the `@taskforcesh/connector-pro` dependency. {% tabs %} {% tab title="Npm" %} ``` $ npm install -S @taskforcesh/connector-pro ``` {% endtab %} {% tab title="Yarn" %} ``` $ yarn add -S @taskforcesh/connector-pro ``` {% endtab %} {% tab title="Pnpm" %} ``` $ pnpm install -S @taskforcesh/connector-pro ``` {% endtab %} {% endtabs %} You can now import the package and instantiate a connector. Notice that you can have as many connectors running in parallel as you want. That will just add redundancy, however once is often enough. ```typescript import { ConnectPro } from "@taskforcesh/connector-pro"; ConnectPro( "My Connection", "My token", // IORedis options { host: "localhost", port: 6379, }, // Options { // Specify a team where to put this connection if needed. team: "my team", // Specify a custom backend if using on-premises backend: "ws://localhost:9876", } ); ``` # On Premises Taskforce.sh is also available On-Premises. Currently, it is supported on the AWS platform via the [Marketplace](https://aws.amazon.com/marketplace/pp/prodview-bvdifzvjsoxsm) and as a [docker image](/on-premises/docker). The On-Premises deployment supports unlimited connections and teams. # AWS ### Requirements The On-Premises deployment is a stateful service that only requires storing some environment variables in AWS System Manager. However, it requires some external services in order to work: * An AWS Application Load balancer with HTTPS enabled for your domain. * A PostgreSQL instance to use for your persistent data, you can provision one in AWS for less than 15$ a month. * An Auth0 application, is required to handle user authentication, the free version is perfectly fine unless you need more advanced features. * A Mailgun account is used to notify users via email, also the free version is enough. Since the deployment is stateless, upgrading to a newer version does not require any special administration other than deploying the latest AMI image available in AWS Marketplace. {% hint style="info" %} Currently, you can only configure one Taskforce.sh instance per region. If you for some reason require more instances per region please contact us. {% endhint %} ### Installation Installing Taskforce.sh in AWS is pretty straightforward, but there are a few steps to follow. The instance has some dependencies and requirements that need to be fulfilled. Please follow the instructions on these pages in order to get your instance ready and running. When the installation is completed you will get 2 services, a Configuration service running on port **8080** (see [Configure your instance](/on-premises/aws/configure-your-instance)), and the dashboard itself that runs on port **4200** (which only will work via HTTPS access due to Auth0 security constraints, see [Setup loadbalancer](/on-premises/aws/setup-loadbalancer) and [Setup route 53](/on-premises/aws/setup-route-53)). # Launch AWS AMI You can find the latest Taskforce.sh AMI on-premise product on the AWS Marketplace. All versions are listed [here](https://aws.amazon.com/marketplace/seller-profile?id=36523b71-72b9-4cc6-a5e6-89c3af6b2719). Simply follow the configuration steps provided by the AWS Marketplace and launch the instance. It is convenient to create a new security group directly on the launch configuration wizard: ![](/files/-MjStCSTSGnG0P7u6SKq) {% hint style="danger" %} Note that by convenience the source IP is 0.0.0.0/0, however you can later restrict sources to narrow ranges, like only the loadbalancer used for routing the traffic to the instance for example. {% endhint %} You can also optionally create the security group after the AMI has been launched: ![](https://gblobscdn.gitbook.com/assets%2F-MG01zoyzHgr18e2IutZ%2F-MgopRqwDR9dNlW4UI9I%2F-MgopfwjLxirSmmywS9i%2Fimage.png?alt=media\&token=2a97f125-78ba-43ed-8a6c-2db1eb346b6a) Create two inbound rules, one for accessing the config tool and the other for accessing the Taskforce.sh dashboard itself. ![](/files/-MgopseenF_78Tw-AR1I) {% hint style="info" %} We are allowing all sources in this guide, but probably you will only want to allow a loadbalancer as source and configure TLS endpoints for production security. {% endhint %} You will also need to manually add an outbound rule, which normally is created automatically if using the wizard:
Finalize by creating the security group. ![](/files/-MgopXnawBqF4xz7MSA3) Go back to your instance and choose "modify security groups": ![](/files/-Mgoqu1ybrP0iOHBBTMb) Add the new security group and save. ![](/files/-Mgoqo13FFMvXVou19EQ) # Create role with SSM permissions Taskforce.sh uses the AWS Systems Manager in order to store the configuration variables of your instance securely and persistently. Your instance will therefore need some permissions to read and write on the Parameter Store. Follow this instructions to create a role with minimal permissions that we will attach to the instance. a) Go to the IAM panel and click on "Create role". ![](/files/-MgLXL5jTdM2hgcnhL5Q) b) Choose AWS Service as the trusted entity and EC2 as Use Case. ![](/files/-MgLXmheG3APAJ5OtEdT) c) Create a new policy: ![](/files/-MgLXyi_57GZTOkrkZB_) d) Choose System Manager as a service: ![](/files/-MgLYDezSvRP8evCFp07) e) For security we will only enable the absolute minimum permissions in the Actions section. So for the "Read" section we will only mark "GetParametersByPath": ![](/files/-MgLZi76i9LN530BmltW) And for the Write section we will mark "PutParameter": ![](/files/-MgLZw0ddmG5vpPlOuj-) f) Restrict access to the Taskforcesh path. Click on "Add Arn" link: ![](/files/-MgL_mUe9LU1GY8ocvxn) And restrict to only paths under "Taskforcesh" and optionally on all regions: ![](/files/-MgLa2qoC5Pc9T01mXLc) Click on "Add" and finalize by clicking on "Next: Tags" and then "Next: Review". g) Review policy and click on "Create policy": ![](/files/-MgLbChPXSwKqJ991a4r) h) Go back to the "Create role" tab (that should still be open) and find the new policy so that we can attach it to the role. Do not forget to "refresh" the table so that the new policy appears: ![](/files/-MgLdQ5hh1cOC8CqH_nW) Continue by clicking on "Next: Tags", then "Next: Review" i) Review and "Create role" ![](/files/-MgLdoSbpKKZTVCtIg4_) # Assign SSM Role When the instance has been launched you will need to assign the role created in previous step a) Go to the EC2 instance panel and choose your instance: ![](/files/-MgLVjglSOSz96Q1YbTm) b) Go to Actions->Security->Modify IAM Role ![](/files/-MgLVwTJub5wLSrv0QYJ) If you are using the new EC2 Dashboard the menus will look like this instead: ![](/files/-MjSKiaCfZZz9DXrbKDD) c) Choose the IAM Role created in previous step and attach it to the instance (do not forget to "refresh" the table): ![](/files/-MgLe4XBXS-8fLu5PWy2) Save to complete the configuration. {% hint style="danger" %} For the instance to be able to use the new role you will need to reboot the instance: {% endhint %} ![](/files/-MjS_HU352_imfJbM0Bg) # Configure your instance In order to configure your On-Premises instance you need to use the configuration tool that is available at your ec2 instance on port **8080**. ![](/files/-MjS6rk-tyOKBeGSyE6T) To login you need to use the username **admin** and your instance id as the **password**. The instance id appears on your EC2 dashboard where the instance is running. {% hint style="info" %} Note that if you access your instance directly from the instance domain, it will use standard HTTP instead of HTTPS. If you want an extra layer of security it is recommended to configure HTTPS by allocating a dedicated subdomain for the admin tool and AWS Certificate manager. These would be the same instructions as to configure the Taskforce.sh, check [Setup loadbalancer](/on-premises/aws/setup-loadbalancer) and [Setup route 53](/on-premises/aws/setup-route-53). {% endhint %} Once you login into the configuration tool you can setup all the necessary settings, starting with PostgreSQL. You must test the connection before you are allowed to save. {% hint style="danger" %} Due to a limitation in the current version, only "postgres" is allowed as the user, this will be improved in next versions. {% endhint %} ![](/files/-MjSXv6Ll8zKCkaMURYN) When all the settings have been set and saved you can also see that they are stored in the Parameter Store: ![](/files/-MjS_0anmUK9RL7vi6Ls) # Create a PostgreSQL instance If you do not already have a PostgreSQL instance that you can reuse you must create a new one. For Taskforce.sh you can use the cheapest available which is the db.t3.micro instance (price between 15$ to 23$ depending on region). ![](/files/-Mi56ApV0N_JRWgXWO9Z) ![](/files/-Mi5Fp4Gxxwc2V0B8w0f) ![](/files/-Mi56Fghq676B-MF-PQI) Magnetic storage and 5Gb will also be enough: ![](/files/-Mi5Lwopp09eOunJHnAc) Specify that we want to create a new security group for this instance: ![](/files/-Mi5G0sQ7dJiSBbXM-uV) In the Additional configuration you need to create an initial database: ![](/files/-Mi5LccC_VCpocRseeTu) Create your database, and now make sure that the security group you created has a inbound rule that gives access to the EC2 instance running Taskforce.sh: ![](/files/-Mi5U6ZhaGqvOfpVvP9S) # Configure Auth0 You need a free Auth0 account to manage your Taskforce.sh users. Login in [auth0](https://auth0.com/) and create a new application: ![](/files/-Mi9fn7MetgZ-0-kPilh) ![](/files/-Mi5d5buR-0XXxXFN9Bh) Choose "Single Page Web Application" ![](/files/-Mi5deDZ_Jt4Xc5sAUvW) Go to the App Settings and configure the callbacks: ![](/files/-Mi5ffXh-9lbLJIR5KoA) Assuming the URI to access your instance is onpremises.taskforce.sh you will need to configure the callbacks like these: ![](/files/-MjXHBHOr3O447gcLL_J) You also need to configure the Allowed Web Origins and Allowed Origins (CORS): ![](/files/-MjXHWw9uKbLidnr3KsU) You can leave the rest of the settings on their defaults. Note that you should pick the "clientId" and the "domain" from this application and use them in your Taskforce instance settings: ![Auth0 Application Settings](/files/vF5GGO42F5h17NAlnf10) ![Taskforce.sh on-premises configuration](/files/RpDsyuJjI5KWcK7DM9ho) # Configure API Besides an application you also need to create an API, just follow this simple steps. Go to APISs: ![](/files/-MiU7S8QcydZJTXd76y5) and create a new API: ![](/files/-MiU7oWqXiKDa-qgZ5kZ) This identifier is the one you need to use on the On Premises configuration tool on the "audience" field: ![](/files/7NC3kbB1nELi0VAx8w6j) ### Troubleshooting If you get login errors, the best way to find the issue is to check on Auth0's Monitor->Logs, where you can see all the calls to Auth0 including the failing ones. If you cannot make sense of this error please contact Taskforce.sh support: . # Configure Mailgun [Mailgun](https://www.mailgun.com/) is required in order to send email notifications from Taskforce.sh to your users. Most likely the free account will be enough for your needs. However Mailgun requires a valid credit card in order to create the account to avoid spammers to exploit their service. You will need to pick the **Private API key** and use it in the Taskforce.sh configuration tool**.** # Setup loadbalancer Taskforce.sh requires a load balancer so that we can enable HTTPS which is mandatory for Auth0. {% hint style="info" %} Note that the screenshots in this page belongs to the old AWS dashboard. {% endhint %} Go to EC2 load balancers and create a new application load balancer: ![](/files/-MhRZ_Q_fEJuO44oNvOa) #### Step 1. Configure the load balancer. Make sure to choose the HTTPS listener: ![](/files/-MhRZuartUmu6do6ODzW) Choose your Availability zones: ![](/files/-Mi4p6gtzcL4phob4b0E) #### Step 2. Configure security settings. In this step you will choose an appropriate TLS certificate that you must have created previously () #### Step 3. Configure security groups. In this step you must choose the security group that you created and assigned to your EC2 instance where Taskforce.sh is running. You may also need to add a security group to make your load balancer accessible. For example for a public-faced Taskforce.sh dashboard you would define a security groups with the following inbound rules:
#### Step 4. Configure Routing. Just leave the settings on this step with its defaults, but port **4200** and just name the target group to something like *taskforcesh-target*. ![](/files/-Mi4re-3r7C-eDmh9jab) #### Step 5. Register targets. Choose the EC2 instance running Taskforce.sh, target port **4200** and click on "Add to registered". ![](/files/-Mi4sIl55HarbtFu_KZc) # Setup route 53 The last step is to configure route 53 so that you can access your instance using your custom domain. #### Create a new record On your desired hosted zone create a new record. Remember that the record needs to match the ACM certificate that you assigned to your load balancer. Choose record type A, and enable Alias, and choose Alias to Application Load Balancer. Finally choose the load balancer created in the previous step: ![](/files/-MhRgeigfJVJOg9XuFqa) Create the record and then you can test it. However it can take up to 24 hours until the domain is available. # Monitors The On Premises instance runs a small Redis instance (128Mb) that is required for running the Monitors in Taskforce's alert system. You can add a local connection for checking that the monitor system is working as it should, like this: ![](/files/0K0gRwFQU6AvfV5trBkT) You will get a new connection that shows the monitor (and other queues as the monitors do their thing): ![](/files/fVJuXPJAnMbUSSaWewXe) {% hint style="info" %} The Redis instance is isolated from the outside and only possible to be accessed from the EC2 host itself, so TLS, nor password authentication is necessary. {% endhint %} # Upgrading From time to time new versions of the On Premises AMI will be released. The upgrade operation requires far less steps than the initial installation. Basically there are 3 things we need to do: * Launch the new AMI reusing the same Security Group as in the old instance (which is already correctly configured) * Assign the same IAM role we used on the previous instance so that the new one can access the parameter store. * Update the Target Group used by the Load Balancer so that we remove the old instance and add the new one. ### Launch the new AMI Since all state required by the service is stored in postgres and in AWS Param Store, we just need to launch the new AMI, but choosing manually the same Security group as we used on the previous version. ![](/files/Bx2RQgxxtm2cufsoflhF) ### Assign the IAM Role Chose the newly created instance and click on the Modify IAM Role option: ![](/files/9cP8lWc3QCZyEYNZx71j) Choose the same IAM Role we used for the old instance: ![](/files/0JvoClLXApCOOCse6CG1) You will need to Reboot your instance so that the new IAM role gains effect. ### Update Target Groups Next, when the instance is up and running we must also update the Target group used by our load balancer so that it points to the new instance. ![](/files/Q8dDYDvJ7wvNWL5YF1nj) Register the new target and wait until it gets "healthy". ![](/files/uy6TnpaTr5rcy72wiE01) You can now deregister the old target so that only the new one will get traffic. You should be able to navigate to the domain that points to the load balancer and test the new instance. Finally pause or terminate the old instance since it is not used anymore. # Troubleshooting As there are quite a lot of steps to get the instance working it is easy that things goes wrong. In this section we will provide some tips in order to help troubleshooting. #### Check logs It is possible to SSH to the instance running Taskforce.sh and use [pm2](https://pm2.keymetrics.io/) in order to get the logs. You can start by running: ``` pm2 list ``` You should get an output like this: ![](/files/vvsv5ujtTAP0CgJEpDmW) Now you can use the id (0 in this case to fetch the logs): ``` pm2 logs 0 --lines 500 ``` # Docker Taskforce.sh On-Premises for Docker The Dashboard is available as a Docker image that can be installed in any container facility that supports docker images. The image provides all the same functionality as the SaaS version available at without any limitations. The image depends on some external services that you need to provide to the container in order to work properly. {% hint style="warning" %} The dashboard should be served from a server supporting https, otherwise, Auth0 will not work. {% endhint %} #### Requirements * A PostgreSQL instance (used for storing all persisted data). * A Redis 7.0+ instance (used for monitoring and internal communication). * [Auth0](https://auth0.com/) credentials (the free version is enough). * [Mailgun](https://www.mailgun.com/) credentials (required for team notifications and alerts. The free version is often enough). # Docker registry The docker image used by the Dashboard is provided by a private docker registry located at In order to install and use the images you will need to subscribe to an On-Premises license on your [subscriptions page](https://taskforce.sh/account/subscriptions).
If you need a trial before you decide to purchase the license, please contact [support](mailto:support@taskforce.sh). When you have a valid license, it will appear in the "Enterprise" section of your [account](https://taskforce.sh/account/enterprise).
You will need this license in order to download and run the image. For this just use the standard docker login and docker pull procedure as shown below: ```bash $ docker login --username=$DOCKER_USER --password=$DOCKER_PASS docker.taskforce.sh $ docker pull docker.taskforce.sh/onprem/dashboard:1.6.0 ``` It is not advised to use the username or the password on the command line, but note that you would need to escape the $ character above: ```bash --username=robot\$5ad883a0-f572-40a0-811a-0d8a5484e331 ``` {% hint style="info" %} The license key will be checked every 15 minutes after the docker container starts, if it fails for more than 5 consecutive attempts, the container will close itself down. {% endhint %} # Environment variables There are a bunch of environment variables that must be set up in order for the instance to run properly. ```properties LICENSE_KEY HOST REDIS_URL POSTGRES_HOST POSTGRES_USER POSTGRES_PORT POSTGRES_DB POSTGRES_PASSWD AUTH0_CLIENTID AUTH0_DOMAIN AUTH0_AUDIENCE MAILGUN_API_KEY MAILGUN_DOMAIN ``` Let's go through these variables in detail. #### License Key The license key to use with this docker container. The container will exit if it is not able to contact and validate the license. You will find your license key on your Taskforce.sh [account](https://taskforce.sh/account/enterprise). #### Host This variable should point to the actual host running the dashboard, for example: #### Redis The dashboard uses Redis for some internal communication and for the Monitoring and Alerts system. A small instance of about 128Mb RAM will be enough in most cases. #### PostgreSQL There are a few variables used to configure the required postgres database, just fill in the values for your database provider accordingly. You won't need a powerful PostgreSQL instance for the dashboard, any minimal instance will do. #### Auth0 The authentication of the users is performed by Auth0. If you are not using Auth0 you can just create a free account which will most likely be enough for your needs. There are some steps involved in configuring Auth0 properly, please refer to this [guide](/on-premises/aws/configure-your-instance/configure-auth0). #### Mailgun Mailgun is used for sending alert messages as well as team invitations. You can get a Mailgun account with generous quotas for free, then just follow these [instructions](/on-premises/aws/configure-your-instance/configure-mailgun). {% hint style="info" %} Currently the dashboard will be exposed at port **9876,** and cannot be configured to a different port. {% endhint %} # Docker compose In order to facilitate the initial installation of the dashboard, please refer to this compose file that you can run locally to check that everything is running, just note that if you run the containers in localhost, you will need to configure [Auth0](/on-premises/aws/configure-your-instance/configure-auth0) accordingly. ```yaml services: backend: image: docker.taskforce.sh/onprem/dashboard:1.0.0 restart: always ports: - "9876:9876" environment: LICENSE_KEY: ${LICENSE_KEY} HOST: ${HOST} POSTGRES_HOST: postgres POSTGRES_PASSWD: ${POSTGRES_PASSWORD} POSTGRES_DB: ${POSTGRES_DB} AUTH0_CLIENTID: ${AUTH0_CLIENTID} AUTH0_DOMAIN: ${AUTH0_DOMAIN} AUTH0_AUDIENCE: ${AUTH0_AUDIENCE} REDIS_URL: redis://redis:6379 MAILGUN_DOMAIN: ${MAILGUN_DOMAIN} MAILGUN_API_KEY: ${MAILGUN_API_KEY} postgres: image: postgres:latest expose: - "5432" restart: always ports: - "5432:5432" environment: POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} POSTGRES_USER: ${POSTGRES_USER} POSTGRES_DB: ${POSTGRES_DB} redis: image: redis:latest restart: always ports: - "6379:6379" ``` As soon as the instance is up and running, you could try to use the [taskforce-connector](/on-premises/using-the-taskforce-connector) to expose the Redis instance that is running in docker to your Taskforce.sh instance. Then you should be able to see the "[monitoring](/on-premises/aws/monitors)" queue that handles the monitor checks and alerts for the rest of your connections. # Using the taskforce connector When your on-premises instance is running, you can connect to your Redis instances either by creating direct connections, but also using the [taskforce-connector.](https://github.com/taskforcesh/taskforce-connector) By default the connector will try to connect to the SaaS version of Taskforce.sh. But when we are running an on-premises instance we need to specify the websocket address hosting the api. For this you should use the "-b" argument. For example, assuming your instance is running at "": ``` taskforce -b wss://my.taskforce.domain -t my-token-xxx ``` # Security primer This section aims to give you some reminders and ideas for best practices for having a tight system security wise. Keep in mind that currently state of affairs is that there are a large number of international leagues with the only purpose of stealing your data, resources and/or money. They have automated tools for discovering any mistake you do, whereas you leave a system open, or publish inadvertently confidential data. {% hint style="danger" %} Do not take security for granted, if you do you are putting your data and assets at high risk. {% endhint %} Credential data in Taskforce.sh is encrypted according to latest cloud standards, as well as the data stored in the backups and persistent redis storage. Still it is important that you follow these steps in order to add another layer of security on top of your queues: * Avoid storing secret/confidential data on the jobs data at all cost. * If you really need to store secret data, always encrypt it using a private key that is not stored on the jobs data. If you need to implement encryption is easier than you think, but it is also easy to implement it wrong, please read this article on how to do it the correct way: # Troubleshooting