On Parallelism and Weirdness

[keethraxx]

есть куча файлов данных и програмка которая что-то с ними делает. какие-то части в основном считают, какие-то в основном читают диск. Чтобы работало быстрее, параллельно работает 12 копий (через GNU Parallel)

Так вот, два варианта

Comments

Faster version still?

[ext_239810]

I am curious: Where did you get the '-j 12' from? Was it a guess or is it based on the number of CPUs? If based on the number of CPUs you should check out -j 150% or -j+4. The first will run 1.5 jobs per CPU core, the second will run (number of cpu cores + 4) jobs in parallel.

I believe you are right that the slowdown is due to reading the uncompressed file from disk. The zcat version will most likely have the uncompressed file in cache and will only need to read the compressed file and will therefore be faster to read.

If myprogram supports reading from stdin or the special file '-', then you can even avoid making the tmp file completely and it will probably be faster still:

ls files/ | parallel -j 12 "zcat files/{} | myprogram -"

Also did you have a chance to watch the intro video to GNU Parallel at http://www.youtube.com/watch?v=OpaiGYxkSuQ

Re: Faster version still?

[keethraxx]

Thanks!!

The -j 12 is based on the number of physical cores (2x6), also (judging by the 'top' output) with 12 jobs it very nearly fills my 24G RAM at the peak with 12 jobs (and X, firefox, etc. running).

-j +0 creates 24 due to HT, I haven't actually tried it for this job due to memory concerns, plus the performance gain from HT is not that huge I think...

Very good point about stdin, I haven't thought about it at all, but it makes a lot of sense. Might be a bit tricky in this case (the job is actually a sas script), but in principle should be doable, I think I'll try it.

actually, I must say I have no idea how pipes actually communicate -- they look very mysterious to me :)

Re: Faster version still?

[keethraxx]

I haven't watched the video yet, just looked at the examples in the documentation, they are quite instructive :)

Actually I first spent some time looking for a job scheduler, because installing smth like Torque for a single user on a single machine is just weird :) and I wasn't really finding anything interesting, probably because I was looking for some form of qsub... and then I thought, what I want should be doable with just a bash script, and there actually are a few versions of that, and I think that's when I found this one which looks nice.