Listing of main services location

User websites
user galleries
mysql DBs
postgresql DBs
forums
ftp server
mrtg
webalizer web sites stats
user ssh

fursuit archive
rss feeds (fpod, archive..)

mail server
nameserver
mailing lists
mysql DBs

File hosting

stream server
ftp user
user ssh

fursuit archive (Round Robin with j2)
stream server (Round Robin with j5)
animal archive
baltosource
roilionsource
Other websites

ftp
mysql DBs
mrtg
webalizer web sites stats
backups

Server Status, Maintenance / Downtime

(dates are based on GMT+2 time)

STATUS DATE TIME DOWNTIME SERVICES DESCRIPTION

DONE 20080915 20:50 00:30 fr filesystem full.
mysql table repairs needed.
Repair done.

DONE 20080715 11:25 00:05x3 j* shpgperproc increase + kernel tuning due to new jail and processes + reboots
DONE 20080611 16:35 03:30 FR* FR provider optical fiber cut. again!
DONE 20080526 21:10 00:15 j1:postgresql postgresql cleanup
DONE 20080526 14:40 00:02 j4 sync data + move to the new server
DONE 20080525 16:31 00:05 j3 sync data + DB + move to the new server
DONE 20080514 11:55 04:20 FR* FR provider optical fiber cut.
DONE 20080426 14:42 00:50 j1 sync data + DB + move to the new server
DONE 20080424 19:10 00:02 j2 sync + move to the new server

DONE 20080330 18:20 00:35 j* HD replacement
DONE 20080330 16:15 02:05 j* One of the new HD is dying.
Data recovery from the dying HD.
HD replacement scheduled
DONE 20080330 14:45 01:30 j* Offline FSCK+Drive testing
DONE 20080330 14:28 00:10 j* Server lock. Rebooting
DONE 20080330 14:00 00:15 j* Server lock. Rebooting
DONE 20080330 13:25 00:20 j* Server lock again..
DONE 20080330 12:40 00:25 j* Disk crash, followed by a server crash.Rebooted

DONE 20080313 21:05 00:03 j* reboot after kernel upgrade
DONE 20080312 11:41 00:03 j* reboot after upgrade
DONE 20080311 11:20 00:05 j1:http,mysql DB migration to new partition
DONE 20080311 10:50 00:07 j1:* Home directories partition migration to new HD.
DONE 20080304 20:10 02:33 j* Memory upgrade + Hard drive4 add-on
DONE 20080304 05:05 01:30 j* Hard drive2 replacement+server swap and reconfigure
DONE 20080112 08:20 00:28 j1:FranceFurs forum phpBB3 migration
DONE 20080109 10:55 00:30 fr:mysqlbaltosource table repairs
DONE 20080109 09:25 00:30 fr:ALL server upgrade + reboot +fsck
DONE 20071003 10:45 00:12 fr:mysql table repair
DONE 20071002 22:00 12:00 fr:ALL local flood on the datacenter network rendered the server almost unreachable
Server got rebooted first because they thought it was a problem on the server.
=> mysql tables need manual repair

DONE 20070920 19:32 2:30 rs:ALL server not responding, ticket open.
took them 2.5 hours to reboot the server, grr.

DONE 20070822 13:29 0:03 fr:ALL reboot after kernel upgrade.

DONE 20070817 17:25 0:12 fr:ALL +1GB RAM addon.
DONE 20070716 12:00 1:30 j*:ALL NIC replacement.
DONE 20070715 13:30 3:00 fr:ALL Data center network down.

DONE 20070709 04:30 3:30 fr:general slowdown The return of the mirroring guy, this time putting the server into swap mode,
rendering it very very slow.
Tweaked the mirroring lock values that were still too high, as he was still hammerring the server
it seemed to work in keeping the server out of swap mode.
Then banned his IP, even though it's a dynamic IP...
DONE 20070706 15:00 3:00 fr:apache heavy mirroring took apache down=>max num of connections per IP decreased
will see if it happens again.
DONE 20070706 13:58 0:20 fr:ALL 160 GB HD replacement by 500GB HD
DONE 20070701 15:30 0:25 baltosource migration to 2nd server
DONE 20070628 ~07:00 ~0:30 j*:ALL Router upgrade
DONE 20070312 11:40 2:00 j*:ALL Server back online in single user mode
All services down, as tech support requires
that I leave the server in single user mode the time they investigate
No investagtion started after more than 2 hours >:|
Can't wait for them forever ! I rebooted in multiuser mode without waiting for them.
Server rebooted ok in multi user mode,
I'll reopen a ticket if the problem comes back, when there is someone available dammit!
DONE 20070312 10:51 0:50 j*:ALL HD3 problem again => reboot
Server, not coming back online. Ticket opened with tech support.
DONE 20070219 16:10 0:25 j1:pgsql forums pgsql php lib problem after upgrade => reinstall
DONE 20070205 15:15 0:05 j*:ALL forced reboot after some HD problem => HD to monitor.
DONE 20061231 15:30 0:10 j1:Mysql Forums repair tables with invalid status
DONE 20061231 14:40 0:35 j1:ALL upgrade j1 system OS (4.11=>5.5)

DONE 20061231 01:15 1:15 j3:ALL upgrade j3 system OS (4.11=>5.5)
DONE 20061123 13:55 3:30 j*:DB Disk full problem
DONE 20061007 13:25 0:05 j*:ALL System upgrade
DONE 20060910 0:30 fr:ALL New 160GB HD upgrade
DONE 20060908 19:55 0:05 j*:ALL old 200GB HD removal
DONE 20060908 19:00 0:05 j*:ALL old 200GB HD transfert
DONE 20060907 09:10 0:30 j*:ALL New 400GB HD install
DONE 20060630 0:02 j2:ALL upgrade j2 system OS (4.11=>5.5)
(server might experience slight slow downs, as I compile the system before switching)
DONE 20060629 0:02 j4:ALL upgrade j4 system OS (4.11=>5.5)
(server might experience slight slow downs, as I compile the system before switching)
DONE 20060628 0:02 j5:ALL upgrade j5 system OS (4.11=>5.5)
(server might experience slight slow downs, as I compile the system before switching)
DONE 20060627 j*:stats webalizer websites stats reinstall
DONE 20060626 09:35 j*:mrtg installed back
DONE 20060625 18:40 0:13 j*:ALL fsck of all the disks,
now that everything should be ok and that the tech won't be touching the server anymore,
I prefer to force a check disk to be sure we start on good base, I think it's better after all the wild reboots from the techs.
DONE 20060625 12:30 05:50 j*:ALL Scheduled HD removal (old system HD)
see the history of problems below to see the stupidity of the guys removing the HD
and for your pleasure here is the best-of
DONE 20060625 15:35 j*:ALL they are removing the HD ..
DONE 20060625 14:30 j*:ALL server back, everything is fine from my point of view, they will try *AGAIN* to remove the old system HD
Hopefully it will work this time and they won't remove the wrong HD.
DONE 20060625 13:52 j*:ALL server now running a check disk, they want me to check the conf before they try *AGAIN* to remove the stupid old HD
DONE 20060625 13:50 j*:ALL I had told them to put back the old drive to check if it was booting back, they said they tried it and it was not working
now I get another message from the person in charge ( not the tech) that he checked himself, and after putting back the old drive
they were supposed to remove it's botting again !!! wtf ! What the heck are they doing
DONE 20060625 13:20 j*:ALL it seems they broke the server again after removing the HD, the server doesn't boot anymore, great
the server was rebooting perfectly before they touched it ! dammit,
they are investigating... great.

DONE 20060625 01:40 xxxx j1,j5:ssh connection fixed
DONE 20060624 21:45 01:05 j*:ALL stupidly locked myself out of the server by inserting the firewall rules,
server is beeing unlocked by tech support..
sorry about that, I think I really need sleep..
DONE 20060624 20:26 0:01 j5:ALL Data sync after reinstall
DONE 20060624 19:52 0:03 j4:ALL Data sync after reinstall
DONE 20060624 19:11 0:00 j2:ALL Data sync after reinstall
DONE 20060624 16:45 0:05 j3:ALL 1 Data sync +config after reinstall
DONE 20060624 15:40 0:10 j1:ALL 1 Data sync +config after reinstall

DONE 20060624 j*:ALL
11:22 All systems should be back to normal now.
I still need to reconfigure and upgrade some of the components, it will go down for a few minutes at a time when I do so.
10:41 main user conf is back, still many things to reconfigure but for the moment websites should be mostly back.
I will have to take them down for a few minutes from time to time to reconfigure / upgrade things.
07:50 system restore finished, I'm reinstalling all
04:39 New ticket for restore open.Great, I have nothing better to do than to wait for them at almost 5am ..
04:35 they still say I have to reopen another ticket for a restore , grrrr 3 hours lost for nothing
03:30 They closed the ticket saying that it was incorrect wtf??
01:35 I've ordered a priority 2 hours restore of the current server,
but it will most likely not help on the reboot problems,
but the server will be back temporarily after I've reconfigured / installed everything back.
00:45 No new server in stock, might take up to 3 days to get a new one
00:10 Bad news, DC told me that the server is not booting anymore.
DONE 20060623 XXXX j*:ALL again and not coming back , DC will check the hardware in single user mode.. might take some time.
DONE 20060623 0:10 j*:ALL reboot again.
DONE 20060622 2:20 j*:ALL reboot still, didn't come back online, manual fsk
DONE 20060622 0:20 j*:ALL scheduled memory + power cables change
DONE 20060622 0:15 j*:ALL again..
DONE 20060622 0:10 j*:ALL again..
DONE 20060621 0:20 j*:ALL server down again, tech investigating...
DONE 20060621 1:10 j*:ALL server rebooted and had to run a fsck
DONE 20060620 0:10 j*:ALL server rebooted again
DONE 20060620 0:20 j*:ALL server rebooted +fsck
DONE 20060325 0:10 j1:php php recompile/upgrade
DONE 20060325 0:02 j1+j3:mysql mysql upgrade
DONE 20060313 0:25 j*:ALL FreeBSD upgrade
DONE 20051008 0:45 j1:phpBB
post&search phpBB problems in the post & search functions with php5.0.5 => required to edit source files of all installations.
DONE 20051008 0:25 j1:php5-modules Scheduled php-modules upgrade
DONE 20051008 0:01 j1:php5 Scheduled php upgrade
DONE 20050905 1:15 j*:ALL Scheduled HD removal, problems after the removal required them to FSCK all left disks (340GB)