Error starting Mysql after power outage

[kuleinc]

I know nothing about mysql. any help would be great. Here is the status I get:

mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-12-26 00:32:19 PST; 5s ago
Process: 2998 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 3006 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Main PID: 3006 (code=exited, status=1/FAILURE)
Status: "Server startup in progress"
Error: 28 (No space left on device)

I also get error: Unable to connect to ZM db.SQLSTATE[HY000] [2002] No such file or directory when trying to connect in web browser. I am accessing server through CLI SSH.

How do I fix this? I'm also getting error when trying to run apt update and upgrade which I think is because of the new version is out? one problem at a time...

[Magic919]

It says no space left on device. Start by looking into that.

[mikb]

One of your file systems is full, or has ended up read-only, or worse, corrupt, after the unclean shutdown.

Code: Select all

df -h

Look for the column that says "Use%" and "100%"

It could be a temporary file/files have been left behind in /var or /tmp, which you will have to find and clean up if ZM/MySQL hasn't noticed and done so.

Do not start running updates/upgrades if your system is in a state, it can only make things worse!

[kuleinc]

Yes I saw that, not sure how to fix it, I'm not super familiar with df-h either:

Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 797M 2.9M 794M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 196G 187G 0 100% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdd2 976M 391M 519M 43% /boot
/dev/loop0 56M 56M 0 100% /snap/core18/2253
/dev/loop1 56M 56M 0 100% /snap/core18/2246
/dev/loop2 62M 62M 0 100% /snap/core20/1270
/dev/loop3 62M 62M 0 100% /snap/core20/1242
/dev/loop4 68M 68M 0 100% /snap/lxd/21835
/dev/loop5 68M 68M 0 100% /snap/lxd/21545
/dev/loop6 44M 44M 0 100% /snap/snapd/14295
/dev/loop7 43M 43M 0 100% /snap/snapd/14066
/dev/sda1 2.7T 673G 1.9T 26% /mnt/83f83fe3-4044-493f-aa74-ed86345d5d9c
/dev/mapper/3TBLVG-FILES3TB 2.7T 89M 2.6T 1% /mnt/434c285e-dafe-4852-98da-9da88c604728
tmpfs 797M 8.0K 797M 1% /run/user/1000


What are these loop entries that are full? it doesn't seem like the RAID array or the system ssd are full... how did / get to 95%?

[kitkat]

how did / get to 95%?

You could try the du command to find out what's using the space:

Code: Select all

sudo du -shc /*

You can then drill down by extending the path to be examined, such as du -shc /var/*

You'll probably get a bunch of "cannot access..." error messages but I tend to ignore those as they're usually for transient files in /proc and /run

[Magic919]

Have a look at logical volumes. It has set aside one for the root file system and it’s stuffed full. Keeping the video storage of ZM off it would be a good start. You could perhaps resize it.

[kuleinc]

The video storage should be on the LVM Array, as its I think three disks, the root file system for the OS should be on its own SSD for speed... ifI recall how I set it up correctly... The LVM is about 3TB or so... specifically for storing the video files from ZM...

joe@camera:~$ sudo du -shc /*
[sudo] password for joe:
0 /bin
388M /boot
4.0K /cdrom
0 /dev
14M /etc
121M /home
0 /lib
0 /lib32
0 /lib64
0 /libx32
16K /lost+found
8.0K /media

Doesn't seem so full unless I'm reading it wrong?

[Magic919]

/dev/mapper/ubuntu--vg-ubuntu--lv 196G 187G 0 100% /

[kuleinc]

if I del whatever is in there will that get it working? is that safe?What is that? Is that camera footage that I somehow didn't setup on the LVM? Sorry I'm real lost here...

[kitkat]

The video storage should be on the LVM Array, as its I think three disks, the root file system for the OS should be on its own SSD for speed... ifI recall how I set it up correctly... The LVM is about 3TB or so... specifically for storing the video files from ZM...

joe@camera:~$ sudo du -shc /*
[sudo] password for joe:
0 /bin
388M /boot
4.0K /cdrom
0 /dev
14M /etc
121M /home
0 /lib
0 /lib32
0 /lib64
0 /libx32
16K /lost+found
8.0K /media

Doesn't seem so full unless I'm reading it wrong?

That looks incomplete to me and I'd expect to see /mnt, /opt, /proc, /root, /run, /sbin, /sys, /tmp, /usr, and /var as well.

[kuleinc]

The video storage should be on the LVM Array, as its I think three disks, the root file system for the OS should be on its own SSD for speed... ifI recall how I set it up correctly... The LVM is about 3TB or so... specifically for storing the video files from ZM...

joe@camera:~$ sudo du -shc /*
[sudo] password for joe:
0 /bin
388M /boot
4.0K /cdrom
0 /dev
14M /etc
121M /home
0 /lib
0 /lib32
0 /lib64
0 /libx32
16K /lost+found
8.0K /media

Doesn't seem so full unless I'm reading it wrong?

That looks incomplete to me and I'd expect to see /mnt, /opt, /proc, /root, /run, /sbin, /sys, /tmp, /usr, and /var as well.

So how to fix it?

[kitkat]

So how to fix it?

Wait until it completes and the command prompt reappears.

It looks like /mnt would be next and you have a large drive with a lot of data mounted there (/dev/sda1 2.7T 673G 1.9T 26% /mnt/83f83fe3-4044-493f-aa74-ed86345d5d9c) which may take a while to calculate.

If it stops by itself at /media then there may well be some sort of file system error.

[kuleinc]

Sorry I didn't realise it takes so long to run the command,it was giving errors:

joe@camera:~$ sudo du -shc /*
[sudo] password for joe:
0 /bin
388M /boot
4.0K /cdrom
0 /dev
14M /etc
121M /home
0 /lib
0 /lib32
0 /lib64
0 /libx32
16K /lost+found
8.0K /media
673G /mnt
4.0K /opt
du: cannot access '/proc/440814/task/440814/fd/4': No such file or directory
du: cannot access '/proc/440814/task/440814/fdinfo/4': No such file or directory
du: cannot access '/proc/440814/fd/3': No such file or directory
du: cannot access '/proc/440814/fdinfo/3': No such file or directory
0 /proc
124K /root
du: cannot access '/run/user/1000/gvfs': Permission denied
2.9M /run
0 /sbin
1.5G /snap
4.0K /srv
4.1G /swap.img
0 /sys
104K /tmp
6.6G /usr
14G /var
699G total

[kitkat]

That looks better

But it doesn't look good

Subtracting the the 673GB on /mnt from the 699GB total leaves only about 26GB used, and as that's nowhere near the supposed 196GB capacity of that volume I'm going to suggest that there's some sort of file system error and that you need to run fsck on it.

I definitely recommend copying as much of the data as possible to another drive before doing anything further.

Unfortunately, this is your file system's root and because you can't (or certainly shouldn't) run fsck on a mounted volume you'll have to boot into some sort of RAM-based rescue system or use other media such as a DVD or USB device (or perhaps attach this drive to another system?)

Once you've done that you can run something like

Code: Select all

fsck -pC0 /dev/mapper/ubuntu--vg-ubuntu--lv

p = attempt safe auto-repairs
C0 = show Counter (that's a zero at the end)

If it says the drive is clean and doesn't need checking then add a -f parameter to Force it.

It may look like it's stopped at times and scary screens of text may fly past but be patient and let it finish

If there are still problems after that then you could leave out the p parameter and tary again, answering any questions manually, or swap the p for y - it's more aggressive than -p and answers Yes to everything (and at this stage there are probably few other options).

If you want to see what's up before committing to anything then leave out both p and y and use n instead for a dry run (answer No to all).

Good luck - And don't forget to backup the data first!

[kuleinc]

sorry it took me so long to try this, I was setting up my Xeon unraid server, I can only work on one server at a time in my workshop...

Heres what happened:

mint@mint:~$ fsck -pC0 /dev/mapper/ubuntu--vg-ubuntu--lv
fsck from util-linux 2.34
fsck.ext2: Permission denied while trying to open /dev/mapper/ubuntu--vg-ubuntu--lv
You must have r/w access to the filesystem or be root
mint@mint:~$ sudo fsck -pC0 /dev/mapper/ubuntu--vg-ubuntu--lv
fsck from util-linux 2.34
/dev/mapper/ubuntu--vg-ubuntu--lv: clean, 283448/13107200 files, 7044855/52428800 blocks
mint@mint:~$