Please enable more threads for process control manual setting ....

Started by mking, November 16, 2013, 12:58:35 AM

Previous topic - Next topic

mking

Hi Mario, I have a new system and wondered why it didn't seem to run iMatch much faster for database access etc.
and then I discovered in preferences/application/process control that you throttle the number of threads, which makes sense for defaults - but why limit manual adjustment to 8 ?
My system barely gets out of bed with only 8 threads... I have 16 cores with 32 hyperthread capability and RAID0 SSD and memory to match.
Please can we have the manual thread limit at least at 32 (for now :)

Txs,
Mike

Mario

Throwing more threads at the problem is not the solution. All threads compete for the hard disk. All threads need to lock the database for short periods of time while writing to it, stalling other threads. More threads means more database locks and more stress on the disk. You may actually get better performance with fewer threads, give it a try!

Having more threads and more CPU's is great when all threads can operate independently from each other, e.g doing calculations in memory, then combining their results. But when IMatch ingests metadata or reads image files to produce thumbnails, cache images, calculates file check sums and the like, all threads compete for the same database (disk I/O and locks), the file system and then hard disk (reading and writing data).
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

mking

Mario, I totally understand what you are saying - but we are now moving beyond mechanical HD limitations with SSD both in terms of raw speed and access time.  I have a RAID0 SSD array capable of delivering 1.2GB/sec. Most new motherboards now at least support 2 x SataIII = 1GB/sec in RAID0 and a couple of cheap SSDs can deliver that along with very very fast access - which is needed as well.

Anyway I did test less threads and its slower - even 8 threads is only sucking 15% of my SSD RAID0 array bandwidth i.e. its sucking only 200mb/sec. and using only 30% of my CPU.

I don't know what overhead the database locks have - but if you let me have more threads I'll let you know which bottleneck I think I hit first.

Txs,
Mike

Mario

Increased limits to 32 threads - for current and future high-end machines this gives a bit more air to play.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook