Compare commits

..

8 Commits

6 changed files with 157 additions and 58 deletions

1
.gitignore vendored
View File

@@ -1 +1,2 @@
venv
docker-compose.dev.yaml

View File

@@ -1,6 +1,7 @@
FROM python:3.14.7-alpine3.24
RUN apk update && \
apk add ipmitool
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1

View File

@@ -1,2 +1,81 @@
# PBSupper
## Description
This program/container automatically powers on a remote server via 623/UDP to iDRAC, and likely other BMCs, connects to Proxmox Backup Server via SSH, so 22/TCP but this can be customized, creates and runs a sync-job, then shuts down PBS.
## Why?
I wanted to automate waking up my server, running a sync-job and shuting down said server. iDRAC6/7 doesn't have that option so I've built one.
## Configurations
Here is a description of the most necessary and all the other environment variables
### Necessary envs
#### BMC envs
* IDRAC -- IP address of the remote BMC **(Default: None)**
* IDRAC_USR -- Username of the remote BMC. This user needs to have permission to power on/off the server as well as check the power status **(Default: None)**
* IDRAC_PWD -- Password of the IDRAC_USR user **(Default: None)**
#### Remote Server envs
* HOST -- IP address, or FQDN, of Proxmox Backup Server. It needs to listen for SSH connections **(Default: None)**
* HOST_USR -- Username of the user on Proxmox Backup Server **(Default: root)**
* HOST_USR_PWD -- Password for the username specified in **HOST_USR** **(Default: None)**
If you want to use SSH-keys, use the following environment variables and leave **HOST_USR_PWD** empty
* HOST_USR_SSHKEY -- Private ssh-key that will be user to connect to PBS. Leave empty if using username+password **(Default: None)**
* HOST_USR_SSHKEY_PASS -- Passphrase for the ssh-key. Leave empty if key is passwordless **(Default: None)**
#### PBS envs
* LOCAL_DATASTORE -- Name of the datastore on the PBS that is passed in **HOST** variable **(Default: None)**
* REMOTE_DATASTORE -- Name of the datastore that the local datastore will sync to/from **(Default: None)**
* REMOTE_PBS -- IP address, or FQDN, of the remote PBS that the local PBS will sync to/from **(Default: None)**
#### Container-specific envs
* RUN_SCHEDULE -- At what time should the server be woken up. Written in Cron notation **(Default: "45 17 * * 5")**
* RUN_SCHEDULE_TZ -- Timezone in which container is. Set to your local timezone as otherwise the wake-up time might be inaccurate. **(Default: "Europe/Stockholm")**
### Other environment variables
Here are the other environment variables that can be set
* HOST_SSH_PORT -- SSH port at which PBS is listening on **(Default: 22)**
* SYNC_DIR -- Direction of the sync-job. **(Default: push)**
* PBS_RATE_OUT -- Speed at which sync is *sent*. Reffer to [PBS documentation](https://pbs.proxmox.com/docs/managing-remotes.html#bandwidth-limit) **(Default: None)**
* PBS_RATE_IN -- Speed at which sync is *received*. Reffer to [PBS documentation](https://pbs.proxmox.com/docs/managing-remotes.html#bandwidth-limit) **(Default: None)**
* PBS_TASK_LIMIT -- How many tasks should be analyzed. Best to leave it at default **(Default: 50)**
* PBS_MAX_RUNTIME_MIN -- Max time which PBS might run after the sync-job. Usually it is caused by Verification jobs or Garbage collection jobs. If a *aptupdate*, *reader* or other tasks are running, then MAX_RUNTIME might be longer as these tasks are critical. **(Default: 180)**
* PBS_MAX_SHUT_MIN -- Max time which PBS has for a safe shutdown. This variable is used after a *shutdown* command is sent to PBS server. **(Default: 10)**
* AUTO_ADD_NAME -- This is a name that is set as sync-job id. **(Default: "autoBackupScript_one-shot")**
* RUN_ONE_SHOT -- If the script should only be ran once. Do not set/use unless you have specific usecase. **(Default: False)**
 
![Yeeee](https://data.shupogaki.org/assets/1/HPDU4TAbEAAzMCj.gif)

View File

@@ -1,18 +1,18 @@
---
# sudo crontab -e
# 0 2 * * * cd /path/to/docker-compose && /usr/bin/docker compose up -d >> /var/log/pbsupper-docker-compose.log 2>&1
services:
pbsupper-1:
container_name: pbsupper
user: 1600:1600
image: tea.shupogaki.org/yuruc3/pbsupper:v0
image: tea.shupogaki.org/yuruc3/pbsupper:v0.2.1
environment:
LOCAL_DATASTORE: somethingSomething
REMOTE_DATASTORE: somethingSomething
REMOTE_PBS: somethingSomething
RUN_SCHEDULE: "45 10 * * SUN"
HOST: somethingSomething
HOST_SSH_PORT: somethingSomething
HOST_USR: somethingSomething

32
main.py
View File

@@ -32,6 +32,7 @@ AUTO_ADD_NAME: Final[str] = os.getenv("AUTO_ADD_NAME", "autoBackupScript_one-sho
RUN_SCHEDULE: Final[str] = os.getenv("RUN_SCHEDULE", "45 17 * * 5")
RUN_SCHEDULE_TZ: Final[str] = os.getenv("RUN_SCHEDULE_TZ", "Europe/Stockholm")
RUN_ONE_SHOT: Final[bool] = (os.getenv("RUN_ONE_SHOT", "false").lower() == "true")
@@ -160,21 +161,21 @@ def theCodeBasically():
# print(epoch_time)
# Check if power is on
PWR_STATE = subprocess.run(["ipmitool", "-I", "lanplus","-H", IDRAC,"-U", IDRAC_USR,"-P", IDRAC_PWD,"chassis", "power", "status",],capture_output=True,text=True,)
PWR_STATE = subprocess.run(["/usr/sbin/ipmitool", "-I", "lanplus","-H", IDRAC,"-U", IDRAC_USR,"-P", IDRAC_PWD,"chassis", "power", "status",],capture_output=True,text=True,)
if PWR_STATE.returncode != 0:
raise RuntimeError(f"ipmitool failed: {PWR_STATE.stderr.strip()}")
raise RuntimeError(f"/usr/sbin/ipmitool failed: {PWR_STATE.stderr.strip()}")
if "off" in PWR_STATE.stdout.lower():
subprocess.run(["ipmitool", "-I", "lanplus","-H", IDRAC,"-U", IDRAC_USR,"-P", IDRAC_PWD,"chassis", "power", "on",],check=True,)
subprocess.run(["/usr/sbin/ipmitool", "-I", "lanplus","-H", IDRAC,"-U", IDRAC_USR,"-P", IDRAC_PWD,"chassis", "power", "on",],check=True,)
# PWR_STATE = subprocess.run(["ipmitool", "-I", "lanplus", "-H", IDRAC, "-U", IDRAC_USR, "-P", IDRAC_PWD, "chassis", "power", "status"], capture_output=True)
# PWR_STATE = subprocess.run(["/usr/sbin/ipmitool", "-I", "lanplus", "-H", IDRAC, "-U", IDRAC_USR, "-P", IDRAC_PWD, "chassis", "power", "status"], capture_output=True)
# try:
# # power on if needed
# if "off" in str(PWR_STATE.stdout.decode()[-4:]):
# subprocess.run(["ipmitool", "-I", "lanplus", "-H", IDRAC, "-U", IDRAC_USR, "-P", IDRAC_PWD, "chassis", "power", "on"]) # , capture_output=True
# subprocess.run(["/usr/sbin/ipmitool", "-I", "lanplus", "-H", IDRAC, "-U", IDRAC_USR, "-P", IDRAC_PWD, "chassis", "power", "on"]) # , capture_output=True
# except Exception as e:
# print("Most likely wrong password, username or hostname has been passed for idrac.\nExiting\n")
# sys.exit(1)
@@ -345,6 +346,10 @@ def theCodeBasically():
(stdin, stdout, stderr) = ssh.exec_command(f"proxmox-backup-manager sync-job run {AUTO_ADD_NAME}")
# (stdin, stdout, stderr) = ssh.exec_command(
# f"nohup proxmox-backup-manager sync-job run {AUTO_ADD_NAME} >/var/log/{AUTO_ADD_NAME}.log 2>&1 </dev/null &"
# )
# ssh.exec_command(f"proxmox-backup-manager sync-job run {AUTO_ADD_NAME}")
# print("here it runs the sync-job")
# print(stdout.read().decode())
@@ -354,6 +359,9 @@ def theCodeBasically():
if syncJobRunExitCode != 0:
raise RuntimeError(f"Sync job failed: {stderr.read().decode()}")
# raise RuntimeError(f"Failed to start sync job: {stderr.read().decode()}")
else:
print("Sync successful")
# Give some time for PBS to "start" the sync-job task
time.sleep(10)
@@ -377,6 +385,11 @@ def theCodeBasically():
while currenttasks:
print("Waiting 1 minute before checking tasks again")
print("Remaining jobs:")
for job in currenttasks:
print(f"Task {job} Amount {currenttasks[job]}")
time.sleep(60)
currenttasks = getCurrentTasks(ssh, limitOfTasks=PBS_TASK_LIMIT)
@@ -384,6 +397,7 @@ def theCodeBasically():
if time.monotonic() - SYNC_JOB_DATE >= (PBS_MAX_RUNTIME_MIN * 60):
# print(SYNC_JOB_DATE - time.monotonic())
# Forcefully skip if these tasks are running. reader task is, I think, PVE backup
# Also otherjob is something else and it is better to wait if there is one otherjob
if (
@@ -412,15 +426,17 @@ def theCodeBasically():
print("PBS is powered off")
else:
subprocess.run(["ipmitool", "-I", "lanplus", "-H", IDRAC, "-U", IDRAC_USR, "-P", IDRAC_PWD, "chassis", "power", "off"])
subprocess.run(["/usr/sbin/ipmitool", "-I", "lanplus", "-H", IDRAC, "-U", IDRAC_USR, "-P", IDRAC_PWD, "chassis", "power", "off"])
# sys.exit(0)
if RUN_ONE_SHOT:
theCodeBasically()
exit(0)
scheduler = BlockingScheduler()
scheduler = BlockingScheduler(timezone=RUN_SCHEDULE_TZ)
scheduler.add_job(
theCodeBasically,
CronTrigger.from_crontab(

View File

@@ -20,7 +20,7 @@ data:
# Alternatively you can also use this option instead of PRIVATE_KEY: 123...
stringData:
PRIVATE_KEY: |
HOST_USR_SSHKEY: |
-----BEGIN OPENSSH PRIVATE KEY-----
123...
-----END OPENSSH PRIVATE KEY-----
@@ -49,10 +49,12 @@ data:
PBS_MAX_RUNTIME_MIN: somethingSomething
PBS_MAX_SHUT_MIN: somethingSomething
AUTO_ADD_NAME: somethingSomething
RUN_SCHEDULE: "45 10 * * SUN"
---
apiVersion: batch/v1
kind: CronJob
apiVersion: apps/v1
kind: Deployment
metadata:
name: pbsupper
namespace: CHANGE_ME
@@ -61,20 +63,20 @@ metadata:
idrac: v7
spec:
schedule: "*/15 * * * *"
timeZone: "Europe/Stockholm"
concurrencyPolicy: Forbid
replicas: 1
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
selector:
matchLabels:
app: pbsupper
template:
metadata:
labels:
app: pbsupper
server: Server-1
idrac: v7
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext:
@@ -83,7 +85,7 @@ spec:
containers:
- name: pbsupper
image: tea.shupogaki.org/yuruc3/pbsupper:v0
image: tea.shupogaki.org/yuruc3/pbsupper:v0.2.1
imagePullPolicy: IfNotPresent
envFrom: