» Site Navigation |
|
|
|
» Stats |
Members: 146,293
Threads: 4,838
Posts: 208,711
Top Poster: zajawa (23,910)
|
| Welcome to our newest member, msgeeks |
|
|
Welcome to the new server |
|
Feb 25, 2020 - 4:55 PM - by Ai
|
Congratulations, you're looking at this on the brand new server... At last!
It has been a bit of a rocky journey! We're using a non-standard installation of Ubuntu using a ZFS filesystem. This is a feature that they're looking to roll out in the next LTS 20.04 in a months time, but not to the level we've taken it! The feature will only allow the use of whole drives, not individual sections of drives like we need. We had no end of issues during the move, firstly the server refused to boot as it didn't want to read the /boot/EFI partition. We eventually fixed that using an EFI Shell, simply accessing the fileset and renaming the directory twice did the trick although we still have no idea why!
Then we got everything running and installed ready for the move and during our tests we found the MySQL on ZFS is a NIGHTMARE. The databases were performing around 6 times slower on a server with no traffic than they were on the old, fully loaded server. We tried every trick in the book to tune this but ultimately failed. In the end we had to wipe the new server and start from scratch. This time we reserved a 1TB partition from each drive and used that to create a software Raid10 EXT4 array to place the MySQL data files on. This worked much better
I've learned a hell of a lot about custom partitioning, and a lot more than I ever wanted about EFI Shells. Ultimately we have ended up with a server that:
- More power processors than before
- Same RAM as before (speed and capacity) - 32GB
- 14TB of RaidZ1 ZFS disk array for server and site data (before we had3TB in Raid1 and a single 3TB backup drive)
- Every site (and a lot of key linux areas) reside of their own ZFS volume which gives us data snapshots (point in time recovery). 1/4 drives can fail with no impact on data and minor impact on speed.
- For the sites and key linux areas, the snapshots run hourly and are stored for 48 hours. Daily are stored for 2 weeks and weekly are stored for a month. If we delete something or it breaks we can recover up it up to a month later.
- MySQL sits on a 2TB Raid10 array meaning we can tolerate 1/4 drives (potentially 2/4 drives if it is the right two!) failing with little impact.
- Backups are made daily consisting of all site data and the databases. These are both stored on the server for immediate use and copied remotely to a backup server in another datacentre for redundancy. Akuma also copies these backups to his home server. This gives us 5 copies of the data - 1. Original Data, 2. ZFS Snapshots, 3. Backups on server, 4. Backups on another server, 5. Backups on Akuma's NAS. The maximum amount of data we stand to lose in the event of a dual drive failure is up to 48 hours. If I get clever enough, I'll also work out how to remotely back up the snapshots to my own NAS's ZFS array with incremental updates on the fly. This will have to wait for another day though!
- The server software is vastly updated, although we're still limited to using an older version of PHP due to incompatibilities with our scripts and newer versions.
A lot of you may be wondering what happened to the planned upgrade to Xenforo. Unfortunately (I'm using this word a lot today!) things did not go smoothly. There were many issues that I was unable to overcome. These include:
- No direct upgrade path for the downloads script - we have 2844 files for download (fiction and videos), importing them manually would be a nightmarish job.
- Multiple issues upgrading from such old scripts that needed work arounds.
- Inability to recreate a lot of the current functionality, specifically the front page.
- Escalating costs for plugins.
- Zenforo 2.X was also a very new release at the time and as such contained many "features".
This is still part of the plan, but it has had to take a backseat to real life issues and time constraints. To get the project rolling again will cost around $300 in licence extension (yes, you have to pay annually for plugin and forum licenses if you want to stay up to date with current software and upgrade) fees. It will also rely on either Akuma being able to dedicate a significant amount of time to 1. Learning the software, and 2. Custom coding the missing functionality; or more money for a freelance coder to do the job. We'll have to wait and see how things pan out as right now donations are only just covering the single server fee, I paid the new server setup fee and the extra month for the old server out of my own pocket.
Remaining issues from the move:
- DiaperedAnime lost around 2 hours of data. Unfortunately I was unable to restore the changes made while we tested it on the new server back to the old server due to non-backwards compatible
... [Read More]
|
1 Reply | 292 Views
|
|
Upcoming Downtime in December |
|
Dec 21, 2019 - 5:12 PM - by Ai
|
Before the 27th December, YaoiAi, all sister sites, and all hosted sites will be taken offline for a period of time to allow us to migrate to new server hardware.
The current server runs:
- Intel Xeon E3-1245
- 32GB DDR3
- 3 X 3TB Enterprise SATA. Two drives in a Raid1 array, one as on server backups
- 2TB CIFS storage box for remote backups
The new server will have:
- Intel® Xeon® E3-1270 v3
- 32GB DDR3
- 4 x 6TB Enterprise SATA. All drives will be configured in a RaidZ1 array running the ZFS file system. Each site will have its own zpool to allow for point in time recovery via the file system snapshots. ZFS will be installed as the root file system.
- 2TB atom server in another data centre for remote backups
From the benchmarks http://cpuboss.com/cpus/Intel-Xeon-E...l-Xeon-E3-1245 the cpu is much more powerful. The ZFS file system will also improve overall security, speed and resiliency.
There is little difference in price between the two server setups so it makes sense to move us at this time to more powerful hardware.
When it is time for the migration, the sites will be disabled and you will see a message explaining about the downtime. It is expected to last no more than 6 hours from start to finish. The most time consuming parts will be taking a brand new backup, transferring the backup files to the new server, then importing the new data.
Once complete, the DNS will be updated and once propagated all sites on the new server will be accessible and open.
|
|
0 Replies | 1,885 Views
|
|
» Recent Threads |
|
No Threads to Display.
|
|
