Getting into TFL

Jim V · July 2, 2002, 05:00 PM

Has anyone else been having problems even getting into TFL? Yesterday, 1 July, I could not even get the "welcome" page to load, 404 errors. Today, for a while, when I clicked "Forums" I got an internal TFL error page. Was it just me and my computer or has there been others that have had problems gettin' here?

Drizzt · July 2, 2002, 05:22 PM

You're definitely not alone. It was happening to me much of yesterday and today.

HarryB · July 2, 2002, 05:33 PM

Me too. Yesterday it was a time out error. Today around 4PM EST I could access the homepage but was told the forums could not be found...

dZ · July 2, 2002, 05:43 PM

there appear to be issues with the SQL database
me thinks the reports of missing posts from last week were an omen of gremlins in the fields

check out the posts in the general forum and you can see where TFL went Bye Bye at 12:01PM today and reappeard at 4:20 PM

yesterday the site went MIA arround 4:30PM and reappeared at 7:20 AM

Didn't Rich say he was heading to the Great Smokey Mountains with a Mac laptop?

Doesn't the site always go wonky when the boss is out gallivanting?

George Hill · July 2, 2002, 07:34 PM

Oops. Our bad. While Rich went into the woods, all the staph was playing Rogue Spear on the system.

Quartus · July 2, 2002, 07:59 PM

Staph? Germ warfare at TFL HQ?

Or are you saying the system is buggy?

No, wait, you were talking about the mods. Are you saying they bug you?

I think I'll bug out.

(It's STAFF, George.

)

**Rich Lucibella** · July 2, 2002, 08:18 PM

OK-
Here's the info dump:
Ever since The Big Hack, we've had low grade problems: Lots of Zombie processes, can't chance remote reboots. As of the past two weeks it's been reaching critical mass.

I tapped member "tyme" for help and he's been hard at work trying to clean up the problems. He's really put in a lot of hours (GRATIS) and has been on it even when I haven't. He deserves a real big "Thanks" for his efforts.

We were waiting to have someone at the Server before trying to install the new changes, but yesterday the Board went down to overload. It recovered after a couple of hours and, today, we attempted an unattended reboot. Took a good hour to come back. After that I needed to repair the database (that's why I moved all the files and y'all got the "Page Not Found" error).....I'm in the mountains with a dial up that disconnects every 20 minutes!

We're not out of the woods yet (original problem on the reboot remains a mystery), but tyme has the system killing Zombies quicker than a coyote on a jack. Memory usage looks better. Even keel.
Rich

Jim V · July 2, 2002, 09:05 PM

So Rich, you are saying while the board has problems, you have the tyme to get it fixed?

(don't hit me!!)

**tyme** · July 2, 2002, 10:04 PM

I don't know about the zombie processes being any better, but the memory usage is much better thanks to the new kernel, and the machine actually boots now (broken nic, apparently; they had that figured out and I didn't believe them).

Apache and modssl were upgraded because of potential security issues with both.

Some mysterious problems, likely caused all the redhat networking setup scripts, with the tfl and swat ip aliases have been worked around.

That's all the news that's fit to print.

TheBluesMan · July 3, 2002, 06:14 AM

Thank you for lending a hand, tyme.

citizen · July 3, 2002, 06:27 AM

Didn't understand a word you said, tyme; but glad you do.

Thanks.

BrokenPaw · July 3, 2002, 11:13 AM

Why is the system generating zombies? I've only ever seen that when a parent process was blocked and not reaping its children...which is usually a sign of something worse.

Or if you're using NFS. Which could (I suppose) qualify as that "something worse"...

I don't know anything about what's going on behind the scenes on TFL's server, but if I can lend a hand, let me know; I'm a UNIX networking software weenie and part-time UNIX admin...

-BP

**Rich Lucibella** · July 3, 2002, 12:20 PM

BP-
tyme can better respond. All I know is that, with the new kernel rebuild, the problem of zombies is pretty much under control. We just took the server off line for a few minutes to configure it to use all the Ram.

Our resident Linux Guru believes that we may not need to up the Ram as the new kernel is better utilizing it.
Rich

**tyme** · July 3, 2002, 12:36 PM

brokenpaw, yeah, but in this case the parent process doesn't actually exist. The parents are T (stopped), but strace/gdb/etc can't attach.

The only mention of a similar problem I've found is at http://www.sysdump.com/kernel/8-28-01/1126.html

On TFL all sorts of processes are not getting reaped; most file commands (ls, mv, chmod...) and some others like gawk. The parent processes just don't seem to be waiting for their children to exit.

The latest redhat glibc packages allegedly fixed some bug of this type, but I'm starting to suspect there's another similar but that's just very rare or they introduced some other bug in the process of fixing it. Either that or Alan managed to sneak in a patch after 2.4.3 from -ac that broke something, and it's never been identified as a problem. Or both.

Oh, yeah, no NFS.

BrokenPaw · July 3, 2002, 12:49 PM

Quote:

The latest redhat glibc packages allegedly fixed some bug of this type, but I'm starting to suspect there's another similar but that's just very rare or they introduced some other bug in the process of fixing it. Either that or Alan managed to sneak in a patch after 2.4.3 from -ac that broke something, and it's never been identified as a problem. Or both.

Lovely. I'm not as up on the 2.4.x kernel series as I am the 2.2.x series. I run a couple of 2.2.18 servers, but they're not loaded anywhere near to the level that TFL is, so I've never really bothered upgrading them.

It's definitely odd that regular processes like ls are behaving as you describe; do they do it all the time, or only when the system is under heavy load?

-BP

**tyme** · July 5, 2002, 11:01 AM

It happens all the time. When I mentioned it on irc, rick van riel claimed he'd never seen that before. There's potentially a way to debug it but it's going to require at least another reboot.

coonan357 · July 6, 2002, 01:33 AM

A headshot will take care of all your zombie problems , just like in the movies ....

July 2, 2002, 05:00 PM	#1
Jim V Senior Member Join Date: January 31, 1999 Location: SE Michigan - USA Posts: 4,038	Getting into TFL Has anyone else been having problems even getting into TFL? Yesterday, 1 July, I could not even get the "welcome" page to load, 404 errors. Today, for a while, when I clicked "Forums" I got an internal TFL error page. Was it just me and my computer or has there been others that have had problems gettin' here? __________________ *MOLON LABE* If it ain't metal, single stack & single action, it ain't a 1911 no matter what it looks like. 1911 Forum THE TUBBY CURMUDGEON

July 2, 2002, 05:22 PM	#2
Drizzt Senior Member Join Date: October 25, 2000 Location: Moscow on the Colorado, TX Posts: 2,560	You're definitely not alone. It was happening to me much of yesterday and today. __________________ "That a free citizen should have to go before a committee, hat in hand, and pray for permission to bear arms - fantastic! Arm your daughter, sir, and pay no attention to petty bureaucrats." Robert Heinlein - Red Planet

July 2, 2002, 05:43 PM	#4
dZ Senior Member Join Date: May 31, 1999 Location: Exiled, Fetid Swamp, DC Posts: 7,548	there appear to be issues with the SQL database me thinks the reports of missing posts from last week were an omen of gremlins in the fields check out the posts in the general forum and you can see where TFL went Bye Bye at 12:01PM today and reappeard at 4:20 PM yesterday the site went MIA arround 4:30PM and reappeared at 7:20 AM Didn't Rich say he was heading to the Great Smokey Mountains with a Mac laptop? Doesn't the site always go wonky when the boss is out gallivanting? __________________ "O tell the Lacedomecians to damn the torpedoes." BOTR, Chapter V: Some Monsters

July 2, 2002, 07:34 PM	#5
George Hill Staff Alumnus Join Date: October 14, 1998 Location: North Carolina Posts: 11,546	Oops. Our bad. While Rich went into the woods, all the staph was playing Rogue Spear on the system. __________________ MAD OGRE

July 2, 2002, 07:59 PM	#6
Quartus Senior Member Join Date: May 8, 2001 Location: Virginia Posts: 3,823	Staph? Germ warfare at TFL HQ? Or are you saying the system is buggy? No, wait, you were talking about the mods. Are you saying they bug you? I think I'll bug out. (It's STAFF, George. ) __________________ . Better to know what you don't know than to think you know what you don't know.

July 2, 2002, 05:33 PM	#3
HarryB Senior Member Join Date: April 4, 1999 Location: Mesa Arizona Posts: 330	Me too. Yesterday it was a time out error. Today around 4PM EST I could access the homepage but was told the forums could not be found...

July 2, 2002, 08:18 PM	#7
Rich Lucibella Staff Join Date: October 6, 1998 Location: South Florida Posts: 10,229	OK- Here's the info dump: Ever since The Big Hack, we've had low grade problems: Lots of Zombie processes, can't chance remote reboots. As of the past two weeks it's been reaching critical mass. I tapped member "tyme" for help and he's been hard at work trying to clean up the problems. He's really put in a lot of hours (*GRATIS) and has been on it even when I haven't. He deserves a real big "Thanks" for his efforts. We were waiting to have someone at the Server before trying to install the new changes, but yesterday the Board went down to overload. It recovered after a couple of hours and, today, we attempted an unattended reboot. Took a good hour to come back. After that I needed to repair the database (that's why I moved all the files and y'all got the "Page Not Found" error).....I'm in the mountains with a dial up that disconnects every 20 minutes! We're not out of the woods yet (original problem on the reboot remains a mystery), but tyme has the system killing Zombies quicker than a coyote on a jack. Memory usage looks better. Even keel. Rich __________________ S.W.A.T. Magazine* Weapons, Training and Tactics for the Real World Join us at TFL or at AR15.com or on Facebook

July 2, 2002, 09:05 PM	#8
Jim V Senior Member Join Date: January 31, 1999 Location: SE Michigan - USA Posts: 4,038	So Rich, you are saying while the board has problems, you have the tyme to get it fixed? (don't hit me!!) __________________ *MOLON LABE* If it ain't metal, single stack & single action, it ain't a 1911 no matter what it looks like. 1911 Forum THE TUBBY CURMUDGEON

July 2, 2002, 10:04 PM	#9
tyme Staff Join Date: October 13, 2001 Posts: 3,355	I don't know about the zombie processes being any better, but the memory usage is much better thanks to the new kernel, and the machine actually boots now (broken nic, apparently; they had that figured out and I didn't believe them). Apache and modssl were upgraded because of potential security issues with both. Some mysterious problems, likely caused all the redhat networking setup scripts, with the tfl and swat ip aliases have been worked around. That's all the news that's fit to print. __________________ “The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...” (blade runner) “Who are you?” “A friend. I'm here to prevent you from making a mistake.” “You have no idea what I'm doing here, friend.” “In specific terms, no, but I swore an oath to protect the world...” (continuum) “It's a goal you won't understand until later. Your job is to make sure he doesn't achieve the goal.” (bsg)

July 3, 2002, 06:14 AM	#10
TheBluesMan Moderator Emeritus Join Date: June 15, 1999 Location: Ohio Posts: 7,558	Thank you for lending a hand, tyme.

July 3, 2002, 06:27 AM	#11
citizen Senior Member Join Date: September 12, 2000 Posts: 796	Didn't understand a word you said, tyme; but glad you do. Thanks. __________________ ILLEGITIMI NON CARBORUNDUM NRA, GOA, AOC Ignorance can be cured, Stupid is forever Life is too short for dial-up

July 3, 2002, 11:13 AM	#12
BrokenPaw Senior Member Join Date: May 1, 2002 Location: Manassas, Virginia Posts: 101	Why is the system generating zombies? I've only ever seen that when a parent process was blocked and not reaping its children...which is usually a sign of something worse. Or if you're using NFS. Which could (I suppose) qualify as that "something worse"... I don't know anything about what's going on behind the scenes on TFL's server, but if I can lend a hand, let me know; I'm a UNIX networking software weenie and part-time UNIX admin... -BP __________________ Seek out wisdom in books, rare manuscripts, and cryptic poems if you will, but seek it also in simple stones and fragile herbs and in the cries of wild birds. Listen to the whisperings of the wind and the roar of water if you would discover magic, for it is here that the old secrets are still preserved. - Ardaynes #38

July 3, 2002, 12:20 PM	#13
Rich Lucibella Staff Join Date: October 6, 1998 Location: South Florida Posts: 10,229	BP- tyme can better respond. All I know is that, with the new kernel rebuild, the problem of zombies is pretty much under control. We just took the server off line for a few minutes to configure it to use all the Ram. Our resident Linux Guru believes that we may not need to up the Ram as the new kernel is better utilizing it. Rich __________________ S.W.A.T. Magazine Weapons, Training and Tactics for the Real World Join us at TFL or at AR15.com or on Facebook

July 3, 2002, 12:36 PM	#14
tyme Staff Join Date: October 13, 2001 Posts: 3,355	brokenpaw, yeah, but in this case the parent process doesn't actually exist. The parents are T (stopped), but strace/gdb/etc can't attach. The only mention of a similar problem I've found is at http://www.sysdump.com/kernel/8-28-01/1126.html On TFL all sorts of processes are not getting reaped; most file commands (ls, mv, chmod...) and some others like gawk. The parent processes just don't seem to be waiting for their children to exit. The latest redhat glibc packages allegedly fixed some bug of this type, but I'm starting to suspect there's another similar but that's just very rare or they introduced some other bug in the process of fixing it. Either that or Alan managed to sneak in a patch after 2.4.3 from -ac that broke something, and it's never been identified as a problem. Or both. Oh, yeah, no NFS. __________________ “The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...” (blade runner) “Who are you?” “A friend. I'm here to prevent you from making a mistake.” “You have no idea what I'm doing here, friend.” “In specific terms, no, but I swore an oath to protect the world...” (continuum) “It's a goal you won't understand until later. Your job is to make sure he doesn't achieve the goal.” (bsg)

July 5, 2002, 11:01 AM	#16
tyme Staff Join Date: October 13, 2001 Posts: 3,355	It happens all the time. When I mentioned it on irc, rick van riel claimed he'd never seen that before. There's potentially a way to debug it but it's going to require at least another reboot. __________________ “The egg hatched...” “...the egg hatched... and a hundred baby spiders came out...” (blade runner) “Who are you?” “A friend. I'm here to prevent you from making a mistake.” “You have no idea what I'm doing here, friend.” “In specific terms, no, but I swore an oath to protect the world...” (continuum) “It's a goal you won't understand until later. Your job is to make sure he doesn't achieve the goal.” (bsg)

July 6, 2002, 01:33 AM	#17
coonan357 Senior Member Join Date: January 26, 2002 Location: in a state of confusion and disbelief ! Posts: 726	A headshot will take care of all your zombie problems , just like in the movies .... __________________ N.R.A. Lifer and proud of it ! You can't argue with a .357 Mag. Well, you can try, but you'll end up bleeding an aweful lot. -GEORGE HILL 4-08-2002 My Coonan is Not For Sale please do not ask!