Strange shell behaviour on RedHat via ssh

oscarBravo

I've shelled into a RedHat server (RH8?) to look at something for a friend. It behaves very strangely when I do some things, in particular pipe anything through "less".

For example, I typed "less /etc/printcap" about ten minutes ago, and it hasn't returned yet. In another ssh session I was running "top" to see if it was maxing the CPUs or anything - it wasn't - but it blocked that session from updating for about two minutes.

I've quit top and typed "ps ax | grep less" to get a pid for the process that's just sitting there, and now that session's just sitting there too. Neither session is responding to Ctrl-C.

Any clue what's going on? Not my server, so not really my problem, but just curious.

rsynnott

Possibly the network's just horribly overloaded.

oscarBravo

Maybe, but I don't think so. The WAN I'm connecting through is certainly fine, and it's hard to imagine what could create that sort of traffic on the remote LAN.

Besides, it's pretty consistent. I can shell in and 'cat' a file with no problem, but as soon as I 'less' it, it b0rks.

nutzboutstuff

if ping times are bad due to whatever, like a bad firewall or router you can get these problems, i've tried to ssh over lines with almost 3 second ping times and its horrid! its easier to ring the person and dictate the commands. just try pinging the box.if its bad ping times then try find the problem. What are you connecting to it with? putty? or from another windows box? try changing you $TERM variable to something else like xterm, or vt220 and see if that helps...

Red Alert

has somebody installed something like SELinux or Systrace on it? Either of them badly set up could probably do that.

oscarBravo

This is getting interesting. I've opened up a couple of sessions to watch what's happening. Watching "top" in one session and typing "less filename" in another, as soon as I start the "less" command the memory usage starts to go through the roof. Once it fills all available RAM and swap, the kernel kills processes at random.

This server gets hit a hell of a lot by hacking attempts on the ssh port. It's entirely possible that it was 0wned at some point in the past. Am I looking at a symptom of an exploit here? Should I back up user files and reinstall RH from scratch?

parasite

run a rootkit checker on it

Khannie

The "less" could be corrupt.

The kernel killing things randomly is standard behaviour when a system starts running out of memory. This happened on a web server I know of before because there was a memory leak in a php script somewhere.

Would you like me to post my less binary from RHEL 3? It may fix things and probably hasn't changed much since RH8.

rsynnott

Ah, what's the machine like, hardware-wise? A bunch of Dells in Trinity went entertainingly mad when their motherboards all started dying; the Windows ones generally just stopped working while the linux ones behaved much as you've described.

oscarBravo

parasite wrote:

run a rootkit checker on it

Did that: all clear. Thanks!

oscarBravo

Khannie wrote:

The "less" could be corrupt.

I suspected that, but I'm not so sure. I ran "up2date less" and it said I already had the latest version, for what that's worth. The binary is dated from January of this year. Also, I've found that if I "cat filename | less" it works perfectly, where "less filename" causes the memory spiral of death.

oscarBravo

rsynnott wrote:

Ah, what's the machine like, hardware-wise? A bunch of Dells in Trinity went entertainingly mad when their motherboards all started dying; the Windows ones generally just stopped working while the linux ones behaved much as you've described.

Hm. It's a shiny new Fujitsu Siemens server with 512M RAM and a 3GHz Xeon processor (I think - it shows up as two CPUs in "top"). The guy I'm doing this for is planning to run a memtest and CPU burn-in from a mindi ISO at the weekend.

Speaking of mindi, I tried to run it to create a recovery backup, and it did exactly the same thing, which tends to divert my suspicions away from the "less" binary.

Keep the suggestions coming, folks!

rsynnott

Def. sounds like hardware dodginess. Do you have physical access to this machine? Does it have a funny on-motherboard RAID system, or something? Or just a funny motherboard chipset? You may need drivers. (Some weirder modern chipsets are fussy like that). Is somebody sitting on the keyboard?

Khannie

Does more work ok?

Want that binary to check?

oscarBravo

rsynnott wrote:

Def. sounds like hardware dodginess.

rsynnott wrote:

Do you have physical access to this machine?

I did yesterday... could do again. It'll be interesting to see the results of the memtest etc.

rsynnott wrote:

Does it have a funny on-motherboard RAID system, or something? Or just a funny motherboard chipset? You may need drivers. (Some weirder modern chipsets are fussy like that).

It's got a SATA RAID setup. I think it's just got two drives set up in a RAID-1 configuration, but I didn't really look. What are you thinking?

rsynnott wrote:

Is somebody sitting on the keyboard?

Wouldn't matter, the keyboard isn't plugged into it as a rule...

oscarBravo

Khannie wrote:

Does more work ok?

Yup, perfectly.

Khannie wrote:

Want that binary to check?

Gwan then: paul (at) westnet (dot) ie. Ta!

rsynnott

oscarBravo wrote:

It's got a SATA RAID setup. I think it's just got two drives set up in a RAID-1 configuration, but I didn't really look. What are you thinking?

What chip? Some (most consumer) so-called hardware RAID isn't really, and causes more trouble that it's worth.

oscarBravo

rsynnott wrote:

What chip? Some (most consumer) so-called hardware RAID isn't really, and causes more trouble that it's worth.

According to lspci, it's an Intel 6300ESB SATA RAID controller, presumably onboard.

I'd agree that it's not true RAID as we know it, and I'd be inclined to use it as a straight disk controller and do the RAID in software. The question is, could it cause the problems we're seeing here?

rsynnott

Potentially, I suppose. I'd have a google for it and see if anyone's having the same problems...

nutzboutstuff

what about boot cd's, can it handle Knoppix alright? i've had machines do funny things because they ram onboard wasn't seated right, which caused intermittant problems, because RAM was incompatible, but not non-functional. I take it the logs don't indicate any strange behaviour when less goes awol?

Some times i just wish i could plug myself into a linux kernel and find out whats happening!