Removing files listed in txt file

Eye · 06-04-2010 03:36PM #1

I have a directory with over 200,000 files in it but some file are old. I also have a table in mysql which keeps a record of every file in that directory:

Example:

DIR = Recordings
FILE = 1234567.gsm

I can run a query on my table to fine specific files and dump that into a text file. For example I might want to find all files put in this directory by john and get a txt file in the following format:

12344.gsm
12466.gsm
15234.gsm

(Above is as much as I can do what I need help on is below)

I now have a file called john.txt with 300 lines in it each line looking like this:

12344.gsm (but different numbers)

These are the file names of all files put in this directory by john and I want to remove them if they are older than 6 months.

I need help writing a shell script that will:

* read the txt file
* One line at a time check if the file is older than 6 months and if so delete it

MacGyver · 06-04-2010 03:40PM

could you query the table for the dates and them "rm < john.txt" may not be possible but would make for a very short script

ravydavygravy · 06-04-2010 03:42PM

As long as the filenames don't contain spaces, something like this:

cd dir_with_all_the_files
for FILENAME in `cat john.txt`
do
    find . -name ${FILENAME} -ctime 180 -exec rm -f {} \;
done

If you want to be a bit more careful about it:

cd dir_with_all_the_files
for FILENAME in `cat john.txt`
do
    find . -name ${FILENAME} -ctime 180 -exec mv -f {} ${FILENAME}.old \;
done

Then manually check that only the files you want to delete have the .old extension - when happy, delete them with:

rm -f *.old

That should do the trick.

Dave

Eye · 06-04-2010 04:09PM

Cheers for the reply's

ravydavygravy,
just wondering if that will remove them one at a time because we ran into problems when removing files where the argument was too long.

ravydavygravy · 07-04-2010 11:48AM

Thats why you use find - it won't have that issue. But yes, the loop will do them one at a time

Khannie · 07-04-2010 11:52AM

Well, in the 2nd example the loop will just create lots of .old ones. The problem will occur when you hit the rm *.old bit. If you try and do that with a very large number of files it will fail because of the very long command line. Better to move them to a specific directory, then remove just that directory.

ravydavygravy · 07-04-2010 06:29PM

Khannie wrote: »

Well, in the 2nd example the loop will just create lots of .old ones. The problem will occur when you hit the rm *.old bit. If you try and do that with a very large number of files it will fail because of the very long command line. Better to move them to a specific directory, then remove just that directory.

What he said :-)

In general: if you get the "too many arguements" error, you can use find to do what you were trying to do instead:

rm *.old

becomes

find . -name *.old -exec rm {} \;

or

mv *.old somedir

becomes

find . -name *.txt -exec mv {} somedir \;

AndrewMc · 08-04-2010 08:33AM

ravydavygravy wrote: »
find . -name *.old -exec rm {} \;

If you use "+" instead of ";" at the end, find will put as many filenames as possible onto the command line, thus running rm a lot less. It'll figure out how many is too many, and run rm more than once if necessary. This can be a substantial time saver.

Khannie · 08-04-2010 09:12AM

AndrewMc wrote: »

If you use "+" instead of ";" at the end, find will put as many filenames as possible onto the command line, thus running rm a lot less. It'll figure out how many is too many, and run rm more than once if necessary. This can be a substantial time saver.

Nice tip!

Eye · 08-04-2010 10:43AM

Thanks for the help and tips guys, much appreciated

Removing files listed in txt file

Comments