Troubleshooting

Quick Diagnostics

#

Start with these before diving into specific sections:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Is the server running?
curl -s http://localhost:8443/api/v1/health
# Expected: {"status": "ok"}

# What version is running?
curl -s http://localhost:8443/api/v1/version
# Expected: {"version": "x.y.z", "name": "TantoC2"}

# Server logs (last 50 lines)
journalctl -u tantoc2 -n 50
# or
docker logs tantoc2 --tail 50

Server Won’t Start

#

Port Already in Use

#

1

OSError: [Errno 98] Address already in use

Find and stop whatever is using port 8443:

1
2
3
4
5
6

ss -tlnp | grep 8443
# or
lsof -i :8443

# Kill the conflicting process or change TANTOC2_PORT
export TANTOC2_PORT=9443

Data Directory Not Writable

#

1

PermissionError: [Errno 13] Permission denied: '/opt/tantoc2/data'

Fix ownership:

1
2

chown -R tantoc2:tantoc2 /opt/tantoc2/data
chmod 700 /opt/tantoc2/data

TLS Certificate Error

#

1

ssl.SSLError: [SSL] PEM lib (_ssl.c:4012)

Verify the cert and key files exist, are readable, and are valid PEM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Check file exists and is readable
ls -la /opt/tantoc2/certs/server.{crt,key}

# Verify the cert is valid PEM
openssl x509 -in /opt/tantoc2/certs/server.crt -noout -text | head -20

# Verify the key matches the cert
openssl x509 -in /opt/tantoc2/certs/server.crt -noout -modulus | md5sum
openssl rsa -in /opt/tantoc2/certs/server.key -noout -modulus | md5sum
# Both md5sums must match

Config File Syntax Error

#

1

yaml.scanner.ScannerError: ...

Validate the config file:

1

python3 -c "import yaml; yaml.safe_load(open('~/.tantoc2/config.yaml'))"

Missing Dependencies

#

Ensure all required wheels are installed:

1
2

pip show tantoc2
pip install dist/*.whl

Agent Not Connecting

#

Work through these checks in order.

1. Verify the Listener is Running

#

1

tantoc2> listeners list

The listener should show status running. If it shows error or is absent, check:

1

tantoc2> listeners status <listener-id>

2. Verify the Listener Port is Reachable

#

From a machine on the same network as the agent (or the target network):

1
2
3
4
5

# For TCP listener
nc -zv <server-or-redirector-ip> <port>

# For HTTP listener
curl -v http://<server-or-redirector-ip>:<port>/

If the port is not reachable:

• Check firewall rules on the server and any redirectors
• Verify the listener is bound to 0.0.0.0 (not just 127.0.0.1)
• Verify the redirector is forwarding correctly

3. Verify the Agent Build Config

#

The agent’s embedded config includes the callback address and the engagement’s cryptographic parameters. Verify the build was created for the correct listener:

1
2

tantoc2> builds list
tantoc2> builds show <build-id>

The callbacks field should point to the correct address:port.

4. Correct Engagement Activated

#

Ensure an engagement is activated and the agent was built for it:

1
2

tantoc2> engagements list
tantoc2> engagements use <engagement-id>

5. Crypto Mismatch

#

If the agent connects but registration fails, look for errors in the server log:

1

journalctl -u tantoc2 | grep -E "registration|crypto|invalid|mismatch"

Common causes:

• Clock drift: Server and agent clocks disagree by more than clock_drift_tolerance (default 300s). Synchronize clocks with NTP.
• Wrong engagement: The agent was built for a different engagement. Rebuild the agent for the current engagement.
• Corrupted build: The agent binary was modified after stamping. Rebuild.

6. Transport Plugin Not Loaded

#

If you get connection refused on a listener port but the listener shows as running, the transport plugin may have failed to load:

1
2

# Check plugin loading in the startup log
journalctl -u tantoc2 | grep -E "plugin|transport|Discovered"

If the transport plugin is missing:

1
2
3
4
5
6

# Verify the plugin is installed
pip show tantoc2-transport-http

# If not installed
pip install tantoc2-transport-http-*.whl
# Then restart the server

Plugin Not Loading

#

Check Plugin Discovery

#

1
2
3
4
5

# Startup log should show discovered plugins
journalctl -u tantoc2 | grep "Discovered"
# Expected lines like:
# INFO tantoc2.plugins - Discovered transport plugin: http
# INFO tantoc2.plugins - Discovered transport plugin: tcp

If a plugin is missing from the discovery output:

1
2
3
4
5
6
7
8
9

# Is it installed?
pip show tantoc2-transport-http

# Does it declare entry points?
python3 -c "
import importlib.metadata
for ep in importlib.metadata.entry_points(group='tantoc2.transports'):
    print(ep)
"

Plugin Wheel Install Failure

#

If a wheel from the plugin inbox fails to install:

1
2
3
4
5

# The server log will show pip output
journalctl -u tantoc2 | grep -A 5 "pip install"

# Manually attempt the install to see errors
pip install /opt/tantoc2/plugin_inbox/plugin.whl

Common causes:

• Python version mismatch (wheel requires Python 3.11, server runs 3.10)
• Missing build dependencies for C extensions
• Wheel is corrupt or truncated

Plugin Crashes on Load

#

1

ERROR tantoc2.plugins - Error loading plugin file: /path/to/plugin.py

Enable debug logging and restart the server to see the full traceback:

1

TANTOC2_LOG_LEVEL=DEBUG tantoc2-server 2>&1 | grep -A 20 "Error loading plugin"

Engagement Won’t Activate

#

Wrong Passphrase

#

1

Error: Invalid passphrase

The passphrase does not match the stored salt. Common causes:

• Typo in the passphrase (copy-paste from password manager is safer than manual entry)
• Wrong passphrase (check if multiple passphrases exist in the vault)

Engagement Database Missing

#

1

FileNotFoundError: Engagement database not found: /opt/tantoc2/data/engagements/...

The database file was deleted or the data directory was moved. Restore from backup:

1
2
3
4
5
6

# Find the engagement record
sqlite3 /opt/tantoc2/data/tantoc2.db \
  "SELECT id, name, db_path FROM engagements;"

# Restore the database file to the expected path
cp /backups/engagement-<uuid>.db <db_path>

Performance Issues

#

Slow Task Processing

#

Symptoms: tasks sit in sent state for longer than the beacon interval.

Causes and fixes:

1. Agent not checking in: See Agent Not Connecting
2. High database load: Check disk I/O:

   1 2	iostat -x 1 10 # Look for high %util on the data directory device

3. Background service overhead: Reduce sweep frequency:

   1 2	bg_dead_agent_interval: 120 bg_stale_task_interval: 600

Slow API Responses

#

1
2
3

# Profile a specific API call
time curl -s "http://localhost:8443/api/v1/agents" \
  -H "Authorization: Bearer $TOKEN"

If responses are consistently slow (>1s), check disk I/O and WAL file size:

1
2
3

ls -lh /opt/tantoc2/data/engagements/*/*.db-wal
# If large (>100MB), force a checkpoint
sqlite3 <db-path> "PRAGMA wal_checkpoint(FULL);"

Database Growth

#

Enable task archival to move completed tasks out of the active table:

1

task_archival_age: 86400  # archive tasks completed >24 hours ago

Or trigger manually:

1
2

curl -X POST "http://localhost:8443/api/v1/engagements/<id>/tasks/archive" \
  -H "Authorization: Bearer $TOKEN"

Authentication Issues

#

Rate Limited

#

1

{"error": "Too many failed login attempts"}

After 5 failed login attempts in 300 seconds, the username is rate-limited. Wait 300 seconds or restart the server (the rate limit state is in-memory).

An admin can reset a user’s password without waiting:

1

tantoc2> operators update <operator-id> --password <new-password>

Token Expired

#

1

{"error": "Invalid or expired token"}

Access tokens expire after 3600 seconds (1 hour). Use the refresh token to get a new pair:

1
2
3

curl -X POST http://localhost:8443/api/v1/auth/refresh \
  -H "Content-Type: application/json" \
  -d '{"refresh_token": "<refresh-token>"}'

If the refresh token is also expired (>24 hours), log in again.

Tokens Lost After Restart

#

Tokens are stored in memory. A server restart invalidates all tokens. All operators must log in again after a restart.

Database Issues

#

SQLite Lock Error

#

1

sqlite3.OperationalError: database is locked

This is unusual with WAL mode enabled. Causes:

• Another process has the database open with a write lock
• The -wal or -shm files are corrupted

1
2
3
4
5
6
7
8

# Check if another process has the file open
lsof | grep tantoc2.db

# Check WAL file integrity
sqlite3 /opt/tantoc2/data/tantoc2.db "PRAGMA integrity_check;"

# Force WAL checkpoint and truncate
sqlite3 /opt/tantoc2/data/tantoc2.db "PRAGMA wal_checkpoint(TRUNCATE);"

Schema Migration Failed

#

If an engagement database schema migration fails during import or activation:

1
2
3
4
5
6

# Check the migration log in the server output
journalctl -u tantoc2 | grep -i "migration"

# The pre-migration backup is at:
# <db_path>.backup.<timestamp>.db
# Restore from it if needed

Agent Shows as Dormant or Dead

#

Agent status transitions are based on missed check-ins relative to beacon interval:

• Dormant: Missed 3x beacon interval
• Dead: Missed 10x beacon interval

Check:

• Is the agent process still running?
• Can the agent reach the listener (network path)?
• Is the listener still running?

Build Fails

#

• Check builds list to verify the agent package is loaded
• Check available templates for your package
• Ensure --kill-date is a future date
• Review teamserver logs for detailed error messages

File Transfer Hash Mismatch

#

If hash_verified is false:

• The file may have been modified during transfer
• Retry the transfer

Log Redaction Hiding Debug Information

#

Temporarily disable:

1

export TANTOC2_LOG_REDACTION_ENABLED=false

Enabling Debug Logging

#

For detailed diagnostics:

1
2
3
4
5
6

# Set in environment
export TANTOC2_LOG_LEVEL=DEBUG
tantoc2-server

# Or set in config
log_level: DEBUG

Debug output includes all HTTP requests and responses, plugin discovery steps, background service execution, and pipeline message processing details.