Fragmented database access speed

[Richard]

The fact that the IMatch 5 database needs the best read/write speeds possible has been discussed often. Most often the recommendation has been to put the IMatch database on a SSD (Solid State Drive) or, second best, a FAST USB-3 thumb drive. All well and good if the user can do one or the other but not everyone can do so. That leaves the option of putting the database on the fastest Hard Disk Drive available.

Any Hard disk is susceptible to becoming fragmented.

From Windows 7 Help:

Improve performance by defragmenting your hard disk

Fragmentation makes your hard disk do extra work that can slow down your computer. Removable storage devices such as USB flash drives can also become fragmented. Disk Defragmenter rearranges fragmented data so your disks and drives can work more efficiently. Disk Defragmenter runs on a schedule, but you can also analyze and defragment your disks and drives manually. To do this, follow these steps:

Picture of a Play buttonGo to the Windows website to watch the video. (1:03)

To defragment your hard disk
Click to open Disk Defragmenter.

Under Current status, select the disk you want to defragment.

To determine if the disk needs to be defragmented or not, click Analyze disk. Administrator permission required If you're prompted for an administrator password or confirmation, type the password or provide confirmation.

Once Windows is finished analyzing the disk, you can check the percentage of fragmentation on the disk in the Last Run column. If the number is above 10%, you should defragment the disk.

Click Defragment disk. Administrator permission required If you're prompted for an administrator password or confirmation, type the password or provide confirmation.

Disk Defragmenter might take from several minutes to a few hours to finish, depending on the size and degree of fragmentation of your hard disk. You can still use your computer during the defragmentation process.

Notes
If the disk is already in exclusive use by another program or is formatted using a file system other than NTFS file system, FAT, or FAT32, it can't be defragmented.

Network locations can't be defragmented.

If a disk that you're expecting to see under Current status is not showing up there, it might be because it contains an error. Try to repair the disk first, then return to Disk Defragmenter to try again. See Check a drive for errors for more information.

This leads me to a question and I hope that some of our gurus can help me here. Let's say that I have a 100 GB hard drive that is only 1% fragmented per Windows. Let's also say none of the files on the hard drive are fragmented at all - except for my IMatch database. That means that my database must be highly fragmented. Mario has that problem covered.

The compact/optimize step is part of the diagnosis because it accesses every part of the database while producing a new, compacted database as output. It is very likely that any physical damage in the database is detected by this step, including damage in spare ranges of the file, index structures, unused segments and the like. I consider this an important step because the other analysis steps will not reveal such problems.
phase.

Once IMatch has compacted and optimized the database, the database will no longer be fragmented. However, I do wonder what Windows will do with the now compact file. Will it write the file to sequential blocks on the HD or will it put bits and pieces into any open blocks and thus fragment what was just defragmented?

I believe one sure way to keep IMatch databases in tip top shape would be to put them in their own partition to ensure that they are always together.

I would appreciate input from those who know and understand hard disks and file allocation better than I do.

[sinus]

I would appreciate input from those who know and understand hard disks and file allocation better than I do.

This is a very good point, Richard, thanks for this.
I am also interested into some answers from "our gurus"

I do not know, if you allow in this context to ask: how can I best find out, what harddisk is the fastes on my system? I can see for example, that both of my HardDisks (HD) has 7200 rotations, so means this, that both HDs are equal fast?
If you do not like this additional question (what I would fully understand), please simply delete it (I believe, you can do this  )

[Richard]

If you do not like this additional question (what I would fully understand), please simply delete it (I believe, you can do this  )

Yes I could delete it but your question  certainly pertains to what I am attempting to learn. Namely: what should users do to get top performance from their IMatch database? Picking the fastest hard drive is part of it. 

[jch2103]

I do not know, if you allow in this context to ask: how can I best find out, what harddisk is the fastes on my system? I can see for example, that both of my HardDisks (HD) has 7200 rotations, so means this, that both HDs are equal fast?

There are several hard disk benchmark programs available, e.g., CrystalDiskMark and HDTune. See https://www.raymond.cc/blog/measure-actual-hard-disk-perfomance-under-windows/ for a description of several of these.

!! Be careful when downloading/installing programs like this: Some will by default install 'extra' (unwanted) programs, like OpenCandy. You need to exercise care to be sure you are installing only what you really want. !!

[Richard]

!! Be careful when downloading/installing programs like this: Some will by default install 'extra' (unwanted) programs

Oh how I hate that. One must read everything to ensure that other applications are not installed.

[joel23]

!! Be careful when downloading/installing programs like this: Some will by default install 'extra' (unwanted) programs

Oh how I hate that. One must read everything to ensure that other applications are not installed.

Use this link for CrystalDiskMark - use the link to the upper right. No AddWare.

[jch2103]

!! Be careful when downloading/installing programs like this: Some will by default install 'extra' (unwanted) programs

Oh how I hate that. One must read everything to ensure that other applications are not installed.

Use this link for CrystalDiskMark - use the link to the upper right. No AddWare.

Not quite!: Clicking on that link starts a download with no options for choosing which install (at least without stopping the automatic download first?). The downloaded program offers to install Tuneup Utilities 2014, unless you click Custom Install and then unclick 'Install Tuneup Utilities 2014'!

To paraphrase a famous quote, eternal vigilance is the price of keeping your computer free of crapware...

[joel23]

!! Be careful when downloading/installing programs like this: Some will by default install 'extra' (unwanted) programs

Oh how I hate that. One must read everything to ensure that other applications are not installed.

Use this link for CrystalDiskMark - use the link to the upper right. No AddWare.

Not quite!: Clicking on that link starts a download with no options for choosing which install (at least without stopping the automatic download first?). The downloaded program offers to install Tuneup Utilities 2014, unless you click Custom Install and then unclick 'Install Tuneup Utilities 2014'!

To paraphrase a famous quote, eternal vigilance is the price of keeping your computer free of crapware...

Upps - to bad. Sorry.
I can't remember anymore if it was there when I installed CDM or if I used Custom Install. But I don't have those Tuneup Utilities here.

[JohnZeman]

I haven't manually defragmented a disk in probably about 10 years.  And I might be wrong but I awhile back I seem to recall Mario saying something to the effect that newer versions of Windows defragment the hard drives automatically in the background, or can be configured to do so.

So just now I went to Computer > D:\ drive (my data drive) and clicked Properties > Tools > Defragment.  Sure enough it's set to auto-defrag and said my hard drive was 0% fragmented (screenshot attached).


[attachment deleted by admin]

[joel23]

I haven't manually defragmented a disk in probably about 10 years.  And I might be wrong but I awhile back I seem to recall Mario saying something to the effect that newer versions of Windows defragment the hard drives automatically in the background, or can be configured to do so.

So just now I went to Computer > D:\ drive (my data drive) and clicked Properties > Tools > Defragment.  Sure enough it's set to auto-defrag and said my hard drive was 0% fragmented (screenshot attached).

Yes, this is true. When you left some idle time to the defrager. It usually runs at night (1:00AM in your case) or when the computer is idle. If some shuts down his computer every night and - when booted - contiguous works on it and/or the computer is highly fragmented, he never will reach the 0% state.

[joel23]

The fact that the IMatch 5 database needs the best read/write speeds possible has been discussed often. Most often the recommendation has been to put the IMatch database on a SSD (Solid State Drive) or, second best, a FAST USB-3 thumb drive.

Hm, why do you believe USB3 is the second best choice?
I read this again and again, but...   Of course this is true when some needs to use an external disk.
A fast SATA-III disk directly connected by a SATA-III port is the fastest some can use - this is of course so, because the USB overhead is missing.

Even when in external USB3 cases fast SATA-III HDDs are built in (usually cheap desktop HDDs are used), USB always had a very high overhead and unlike with USB1/2, USB3 comes (like SATA) whit an additional 8b10b-encoding ("when 8 bit data have to be read, transport 10 bits" = -20%) with reduces the nominal 5Gb/s speed down to an effective of 4Gb/s. And the the overall overhead brings it down to real ~340MB/s.
Even when this is enough to feed ONE SATA-III disk, is USB shared per controller, so with two real fast disks connected to two ports on one controller, some might reach the edge. I even see NAS put on the market with three or more disks been propagated for using it as fast RAID 0, connected by an USB3 connection. LOL.

This leads me to a question and I hope that some of our gurus can help me here. Let's say that I have a 100 GB hard drive that is only 1% fragmented per Windows. Let's also say none of the files on the hard drive are fragmented at all - except for my IMatch database. That means that my database must be highly fragmented. Mario has that problem covered.

Once IMatch has compacted and optimized the database, the database will no longer be fragmented. However, I do wonder what Windows will do with the now compact file. Will it write the file to sequential blocks on the HD or will it put bits and pieces into any open blocks and thus fragment what was just defragmented?

When Mario talks about DB fragmentation, I am not sure if he means internal fragmentation (eliminating holes within the DB) or external, rearranging the file on disk. Of course: if the DB was shrinked it might take contiguous clusters on the disk (if there are one free), but I believe IMatch does not try to rearrange cluster on the disk to make produce a contiguous file space. Otherwise a compact would take several (half) hours on disks which are highly fragmented AND have a high fill-grade.
I might be wrong here because my disk never have a fragmentation rate higher than 10% (except the SSD) - will do some test later with a new DB.

Some general matters to think about:
For large files a HDD partition should be formatted with the larges possible Allocation Unit Size. Windows use 4KB per default - the largest possible AUS size is 16 times larger (64KB). A larger AUS means, that even when the disk is highly fragmented, always 64KB (16 times more) of data contiguous is read in once before the read/write head has to move to the next cluster.

For Example: a file large as 100.000KB hast 25.000 cluster a' 4KB, but only 1566 cluster a' 64KB (always a full cluster is used, even when only some bytes are written to it, that's why 1566 instead of 1565 cluster)
So let's assume the small DB of 100MB is not fragmented. As soon 1 Byte (say a small keyword) is written to it, another cluster of 4KB is used somewhere on the disk. Might be on the other end of it.  In case of 64KB cluster size, this 1 Byte is written to the last cluster already used, since there is some space left.

The down side of large AUS size: every file <64Kb also exclusively takes exactly one cluster. So every 8KB XMP sidecar wastes 56KB - but this is something I can live with. On the other hand the XMP is large enough to get fragmented over two 4KB clusters otherwise. See the 1# attachment (size on disk)
Another down side is: 64KB cluster disk IMHO can't be defragmented by W7 means a standalone defragmentation utility has to be used.

Additional fill-grade matters, also for SSD. All my disk a not filled more than 77%.

But the DB is IMHO not the real problem.
Since Exiftool has the unpleasant characteristic to store a temp-file as big as the original for every file which was changed in the same directory, fragmentation is done every time when metadata is changed. Even when only a single keyword has been changed.
Subsequently, after writing the new (temp-)file, the original is deleted and the temp-file is renamed to the name of the original file. The new file now is a bit larger and fills totally different clusters and in the end the disk looks like a Swiss cheese.

For me adding one (!) keyword to 1,000 PSD means writing an average of 200GB on temp-files.  Which is a different number than the few MB which are written to the DB at the same time (peanuts). This may vary with more files and a larger DB, but well...

See attachment #2, the disk after 15GB (22 PSD) have been written to it.
Attachment #3 when one keyword has been written to each of the 22 files.
Attachment #4 when all keywords were again removed.
One block = 128 or more 64KB cluster. If using a different scale in the GUI, fragmentation looks worse.
See attachment #5 which is the same as #4 but at a different scale. The contiguous blue boxes to the left represents my DB.

No performance impact yet, but let the fill-grade be 85% and a cluster size of 4KB and imagine the OS have written other files criss cross in mean time (OS temp-file, IMatchs' temp-file, etc.) as it is on notebooks with usually one HDD and one partition, C: always formatted with 4KB AUS.

For my pretty small DB of 2.8GB (22.400 files) it needs 7 seconds to load from my fastest HDD (CrystalDiskMark says 144MB/s r/w) and almost the same when loaded from SSD or from a 4GB RAM-Disk.
So more important IMHO is where the image files resists, which should be the fastest disk, but not necessary a SSD.

[attachment deleted by admin]

[joel23]

Once IMatch has compacted and optimized the database, the database will no longer be fragmented. However, I do wonder what Windows will do with the now compact file. Will it write the file to sequential blocks on the HD or will it put bits and pieces into any open blocks and thus fragment what was just defragmented?

So Richard, I made my tests about DB fragmentation (external, on disk). I deliberately fragmented my DB to 500 fragments on disk (there are tools for it) and after that compact and optimized it. The red boxes in the attachments.
Aside that it took one minute longer, it is as I thought: same 500 fragments.


[attachment deleted by admin]

[Ferdinand]

For file fragmentation, I've found Windows own defragmenter to be conservative and not removing all fragmentation.  For many years I have used PerfectDisk.  The version that I'm using currently has a minimal defrag for SSDs, which a good SSD should be able to tolerate occasionally. 

The problem with full defragmentation is that it increases the size of backups created by software that does incremental, sector-based backups.  Hence I only do it when I am about to start a new external backup disk.

[Richard]

So just now I went to Computer > D:\ drive (my data drive) and clicked Properties > Tools > Defragment.  Sure enough it's set to auto-defrag and said my hard drive was 0% fragmented (screenshot attached).

Earlier today I also checked and it showed 0 % but just out of curiosity I ran defrag anyway. Not once but four times and every time it found work to do. If it was truly 0% I would expect to be told that defrag had nothing to do.

[khfnet]

At the beginning of the post I was interested.
But now I can't / won't follow.
Befor use IMatch5 my PC hast to be tuned?
Please take a look at at the IMatch5 website, the first orientation for buyer:

photools.com => Produkte => IMatch5 => Hardware and Software Requirements :


"IMatch 3 ?  (should be modified)  Hardware and Software Requirements


Minimal Configuration

    PC with a Pentium-1 GHz or higher processor (or compatible)
    Microsoft Windows XP with Service Pack 3, Windows Vista, Windows 7 or later (both 32 and 64 bit supported)
    1 GB RAM
    Hard disk space: 60 MB for the IMatch software and accompanying files, several free gigabytes for the IMatch database and cache files
    1024 x 600 screen resolution minimum
    Compatible Mouse or other pointing device

For larger databases with 100,000 or more images,  2GB RAM  and a hard disk with at least 50 GB suggested."


mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm

There should be clear information regarding the required speed off HDD
a) for DB
b) for files storage

mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm


Is there anyone with comparable HDD speed, and all work fine?
Win7/64 -i7 - 8GB RAM
DB at C: defragted,(also DB), exception for virus scan
Files at NAS. For Test 2800 jpg. After buy 30.000 mixed.
See benchmark

PS
I Think the benchmark do not work well.
Now i tested by WLAN (130Mbits)
See third benchmark

Doesn't it work well on NAS?
I use CrystalDiskMark

[attachment deleted by admin]

[Richard]

Once IMatch has compacted and optimized the database, the database will no longer be fragmented. However, I do wonder what Windows will do with the now compact file. Will it write the file to sequential blocks on the HD or will it put bits and pieces into any open blocks and thus fragment what was just defragmented?

So Richard, I made my tests about DB fragmentation (external, on disk). I deliberately fragmented my DB to 500 fragments on disk (there are tools for it) and after that compact and optimized it. The red boxes in the attachments.
Aside that it took one minute longer, it is as I thought: same 500 fragments.

That tells me that my fear was correct. IMatch rewrites the database fresh but the new file gets written back in pieces and is thus fragmented again.

[khfnet]

But then everybody must have problems.
Ist That so important?

[Richard]

There should be clear information regarding the required speed off HDD
a) for DB
b) for files storage

I wonder how many computer users even know the speed of their HDD.

My computer is also Win7/64 -i7 - 8GB RAM and works fine for IMatch 5 but from the beginning I had a fast USB-3 stick for the database. I had no knowledge of how having the database on my HDD might affect speed. SO I moved the database to my HDD and could not tell that it made a big difference. However, my database is quite small.

[Richard]

But then everybody must have problems.

I don't see that as being the case for everybody but it is the case for some and I started this thread trying to find answers for those who do experience speed problems but can not add SSD or USB-3.

[sinus]

I used this CrystalDiskMark, what Joerg did recommend.
My DB says, 0% defrag.

I use OS : Windows 7 Home Premium SP1 [6.1 Build 7601] (x64)

First image in the row is C, where IMatch is installed
Second image in the row is F, where all my real images are stored.

These are two different harddisks.
Interesting is, that D, the third image in the row, is the same harddisk like C, and the benchmark show remarkable difference (D is slower).
I do not know why, maybe D is on the outside of the circle (well,  simply a guess).
And on ths D the DB of IMatch is stored.



[attachment deleted by admin]

[joel23]

At the beginning of the post I was interested.
But now I can't / won't follow.
Befor use IMatch5 my PC hast to be tuned?

No. No one has said that. But of course everything should at least within the possible limits, well configured and maintained.

Is there anyone with comparable HDD speed, and all work fine?

We can't compare a workstation with a notebook.  And we shouldn't worry about 10MB/s more or less - today CDM shows 133/131MB/s, last week 144MB/s and HD Tune today 128/126MB/s.
Your HDD data is okay for a notebook.

As said in the other thready, with <3.000 files you shouldn't have any problems. Have you ever copied the images to the HDD and relocated the folder in Imatch, to see if the NAS is the source of the problem? Or even better, copy the images to HDD, make a new DB without blinkyblinky (no relations, no buddy files, etc.) and check if it behaves the same.
And are all task done in Imatch? Will say are all images are ingested and are all data written back? Otherwise a test makes no sense.

I Think the benchmark do not work well.

Now i tested by WLAN (130Mbits)
See third benchmark

Doesn't it work well on NAS?

Might be a memory effect. Try again when you have written a GB of data to the NAS. I will try it later on my NAS.

When I told you to connect the NAS to Ethernet instead to 130Mb/s WLAN, than I of course to a 1Gb/s Ethernet. Notebook and NAS are able to run 1Gb right? So why pushing it down?
With 100Mb/s and with 130Mb/s you won't get more than 10-16MB/s (nominal).

[Richard]

Interesting is, that D, the third image in the row, is the same harddisk like C, and the benchmark show remarkable difference (D is slower).
I do not know why, maybe D is on the outside of the circle (well,  simply a guess).
And on ths D the DB of IMatch is stored.

Hi Markus,

I am confused. If I understand correctly D is a partition on the same HDD as C. If that is correct, I don't understand how one can be different from the other.

It is the "outside of the circle" that travels faster and thus is the fastest for read/ write but regardless, it is all one drive. 

[sinus]

Hi Richard
yes, I can understand your puzzleness. I am confused too.

It is really the same drive.
See attachement 1 ... and grrrr, do not look at the red F, I had trouble with this (new) harddisk, and here Paragon Partition Manger responds with an "3725.8 GB Ungültig" (not guilty), but windows does not show this and I can work without problems with this drive F.

And you can see, C and D (is the drive, with what the computer came) is one drive.

I have repeated the test, you can see this in attachement 2 and 3.

Sometimes computer are a miracle! 

[attachment deleted by admin]

[Richard]

Sometimes computer are a miracle!

Indeed! When I think about how someone like Mario must be able to write instructions in languages like C++ and that the computer must convert those instructions to machine language (Ones and Zeros) to turn switches (transistors) on or off, it is amazing that computers even work.

I know of two men who, fifty years ago, wrote computer programs in machine language. I suppose it is not much different than being able to understand Morris Code but...

Then consider how many computer languages Mario knows. He is indeed multilingual.

[RalfC]

I am confused. If I understand correctly D is a partition on the same HDD as C. If that is correct, I don't understand how one can be different from the other.

It is the "outside of the circle" that travels faster and thus is the fastest for read/ write but regardless, it is all one drive. 

It is completly logic, if you speak the right language. 
It is just the way how (modern) harddisks work.

Let's look at it (I'll just write about reading but writing is essentially the same story):

A harddisk has 1 or more physical disks and for each surface a Read/Write-Head. Those heads are always posititioned simultaneously. Each such a position is called cylinder (you referred to it as circle)
When the harddisk tries to read data the heads need to be positioned to the correct cylinder. The time for doing so is called the "seek time" which is an average time to position the heads to the right cylinder.
Then, the harddisk has to start to look at the data and wait for the needed data to pass by. That time is the latency time and it is related to the rpms of the disk.
It also takes some time to switch between the different heads but this time we can count into the latency time.

On the outer cylinders of the harddisk more data can be stored, thus reading from those cylinders is more efficient because when the hd make one revolution (needing 1/rpm) more data is available to be transferred.

To make things a bit more complicated, hard disks have also a cache memory (which varies). This cache is used by the harddisk to buffer data which is stored sequentially afer the data looked for. (That is the cause for faster sequential reads). When doing random reads, this cache memory is not of such a big use, thus the read speed goes down. Which also can be seen from the shown benchmarks.

An SSD does not have any mechanical rotating items, therefore it reads all it data at the same speed (and faster than a harddisk but depending on the logic of the SSD there are differences).

The so far mentioned factors are inside of the diskdrive but the drive also needs to be connected to the computer somehow. And also those interfaces have different specifications how fast they can transmit data:

For the nowadays most common computer internal interface SATA, there are three different max speeds.

• SATA-6G (or SATA III): max 6Gbit/s or 600MByte/s theoretical transmission speeds (the fastest SSDs reach that value but no harddisk)
• SATA-3G (or SATA II): max 3Gbit/s or 300MByte/s theoretical transmission speeds
• SATA: max 1.5Gbit/s or 150MByte/s theoretical transmission speeds

The there are the external harddisks having USB

• USB 3.0 , "Superspeed": 5 Gbit/s (or 625 MByte/s) theoretical, even if only half of the theoretical speed is reached, it is stiill sufficiently fast for the full performance of a harddisk
• USB 2.0, "Hi-Speed": 480 Mbit/s (or 60 MByte/s) theoretical, in practice ~30MByte/s are reached

or NAS connected via different networks (theoretical maximum speed), e.g.:

• LAN: 100Mbit/s (~12 MByte/s)
• Gigabit-LAN: 1Gbit/s (~120 MByte/s)
• WLAN: 54MBit/s (~6 MByte/s)
• WLAN: 130MBit/s (~16 MByte/s)

If one wants to look at the complete performance chain, there are still other factors which do have an influence [without specific order]:

• how often the harddisk needs to try to read the data without errors
• System setup: eg. certain RAID levels may improve the reading speed (or, to some extent, security against Harddisk failure. Note this is NOT the same as a backup!)
• System configuration: Most modern drives would be able to sort the read-commands in such way that the data can be fetched in an optimized way but this also requires a special BIOS setting and a special driver from Windows.
• Fragmentation of the files [which results in many head relocations] but Windows Defragmentation tries to move the most used files to the beginning of the disk (to speed things up)
• other programs trying to access the same hard disk, e.g. Windows Virtual memory [which would be a bad idea to switch off], virus-scanner making a full scan of the harddisk

Btw., if one looks/tests at the transmissions speeds (esp. for a NAS), one needs to be careful, that one does not only test the buffers which can deliver data faster than the actual harddisk (and transmission rates above the theroretical maximum of the interface/media are a good indicator...)

Regards,
Ralf

PS: When I changed the harddisk in my laptop to an SSD, I mounted the original harddisk to a housing with USB3.0 and compared the performance (internal vs. USB3.0) and the result was identical. I.e. a very fast USB3.0 stick may be faster than an internal harddisk.

PPS: sorry for the long post

[joel23]

Hi Markus & Richard,

Interesting is, that D, the third image in the row, is the same harddisk like C, and the benchmark show remarkable difference (D is slower).
I do not know why, maybe D is on the outside of the circle (well,  simply a guess).
And on ths D the DB of IMatch is stored.

I am confused. If I understand correctly D is a partition on the same HDD as C. If that is correct, I don't understand how one can be different from the other.

It is the "outside of the circle" that travels faster and thus is the fastest for read/ write but regardless, it is all one drive. 

It's the same on my other computer (I don't have more than one partition per disk at my main computer, so I never notice). 89MB/s for C: 100GB and 69MB/s D: 51GB
I made a third partition 600MB large. See the results   Guess I should better stick to HD Tune.

Anyway. Hm, a certain bit on the outer cylinder travels faster than a bit on an inner cylinder, but the outer bit has to travel a longer way Both bits have to spin one round to appear again at the same position, what ideally is just beneath the read/write head.
The reason why a disk has a slower transfer rate for middle and inner cylinders is the so called Zone Bit Recording Please read it yourself, its English is better than mine.

Personally I see no need for having more than one partition on one disk, except for using them with different cluster size. But than (if possible) better use another disk(s) and don't fill them to much. One SATA-III port should be able to feed 3-4 cheap HDDs (a' 90-100MB/s - with a multiplier) without a problem and in addition with good defragmentation tool, the most used files are arranged at the outer cylinders.
Even older SATA-II disks can still serve pretty fast, see the first one in the attachment (outer, middle, inner transfer rates), which is my D: (User Profiles/Documents)
The other lame once are good for data which don't need to be backuped: Google Drive, Copy.Com etc. pp.  Or if some has old SATA-II left, use three or four of them together in a RAID 0, which are good for ~200-270MB/s (OS and Photoshops temp-file, etc.)

Talking about loading the DB from an USB stick: it takes double as long (15s USB-II) as from my HDD. But than it takes much longer to pull the thumbs out of it and of course also writing metadata takes much longer and often the screen switched to white on a slower computer (only 4GB RAM)

[attachment deleted by admin]

[sinus]

PPS: sorry for the long post

Hi Ralf

Nothing to say sorry, really not! Your post was very interesting, now I do know again more ... I wonder, how long I can remember  I hope, the harddisk chrashes before my brain will forget it 

[sinus]

Thanks, Joerg, for your posting.
Together with Ralf's posting, I had now a very good harddisk-lesson! 

BTW: why is in your second test the write speed so high? I wonder, if this is for real or have I overlooked something.

[joel23]

Thanks, Joerg, for your posting.
Together with Ralf's posting, I had now a very good harddisk-lesson! 

BTW: why is in your second test the write speed so high? I wonder, if this is for real or have I overlooked something.

Well, seems CrystalDiskMark got irritated by the small partition size. That's why I wrote "I better should stick to HD Tune"

[cytochrome]

Out of curiosity I downloaded CrystalDiskMark and ran the benchmark for the three disks I use with IMatch:

- my C: with IM is a low end 120Go SSD (Kingston V300 Sata 3)
- Disk F: with the image files is a 7200 rpm 3 To Barracuda
- Disk K: is a Sandisk extreme 64Go SB 3 stick for the DB and the cache

But now I don't know what to do with the result!! I don't see where the bottle necks could be, not that I witness any, the system is relatively fluid with IM and about 60000 files.

Francis

[attachment deleted by admin]

[joel23]

Out of curiosity I downloaded CrystalDiskMark and ran the benchmark for the three disks I use with IMatch:

- my C: with IM is a low end 120Go SSD (Kingston V300 Sata 3)
- Disk F: with the image files is a 7200 rpm 3 To Barracuda
- Disk K: is a Sandisk extreme 64Go SB 3 stick for the DB and the cache

But now I don't know what to do with the result!! I don't see where the bottle necks could be, not that I witness any, the system is relatively fluid with IM and about 60000 files.

Francis

This looks good - except for the SSD. My low end 120GB Samsung 840 does ~470/134MB/s and by this it's slower than my HDD on write.
I would say there is no real bottleneck related to disks!
Seems this thread is driving us nuts?

ps
Except that the SSD is a bit full - 10% left is a bit tight.

[attachment deleted by admin]

[sinus]

Should be forbidden - such quick disks! ... except it would be mine! 

- joke off -

[cytochrome]

Yes, it drives us nuts... its always the same, when a benchmark appears on a forum everybody wants to measure it's system performance, sort of Kindergarten play, my dad has a bigger car and so on.

Yes my C: SSD is full, I delete often tmp files and abandoned programs. I plan to change it for "bigger" one. Yours is in better shape.

Francis

[joel23]

Yes, it drives us nuts... its always the same, when a benchmark appears on a forum everybody wants to measure it's system performance, sort of Kindergarten play, my dad has a bigger car and so on.

Right If it helps I do again: I believe you are fine.

Yes my C: SSD is full, I delete often tmp files and abandoned programs. I plan to change it for "bigger" one. Yours is in better shape.

Francis

I just moved all user files to D:, a HDD. Especially iTunes (movies/music) took a lot of space. Bridges cache is per default created in there and IMatchs cache as well if some don't take care. I have no pagefile at all and temp is redirected to a RAM-Disk. I deleted all .dmp (dumps) which are created once in a while etc. Saves a lot of space.

[Richard]

Seems this thread is driving us nuts?

True but it is also providing a lot of good information for readers. Hopefully those who read it will be able to separate the wheat from the chaff and put some facts to good use so that they get the best performance possible from their computer(s). YMMV (Your Mileage May Vary)

[joel23]

Seems this thread is driving us nuts?

YMMV (Your Mileage May Vary)

Of course. Always.
Important is IMHO to tell what users can expect, not what specifications and marketing are want them to believe / are telling them.

[jch2103]

ps
Except that the SSD is a bit full - 10% left is a bit tight.

Yes, that's extremely tight. The folks at anandtech.com suggest as a general rule leaving at least 25% free on an SSD (unless the manufacturer has set aside a similar amount of over-provisioned space). This greatly improves performance consistency. See http://www.anandtech.com/show/6489/playing-with-op for more technical details (note that they use a Samsung 840 as one of their examples). Also see http://www.anandtech.com/show/6433/intel-ssd-dc-s3700-200gb-review/3 regarding consistency issues.

I came to the conclusion I needed a 256GB SSD for my OS/Windows/programs disk that I usually keep half full.

[Gerd]

Hi,

regarding to yout tests I have also tested my drives (Notebook). C: is the default harddisk (5.600 rpm), where most of my pics are stored, F: is the second (build-in) drive with a Samsung EVO 840 SSD (256 GB), where I have 50 GB free. Here I have only strored my IM-db (5,2 GB). I need the SSD for my big Excel-sheets, not for other things.

But the speed of my SSD looks rather quick  , Samsung delivers an own check-pgm (Screenshot2):



[attachment deleted by admin]

[sinus]

Phew, Gerd, I do not know, but this test (with red marked) is not possible, is it?

[Gerd]

May be, it depends on the typ of connection ... it's a 6Gb/s SATA3 connection ...

[joel23]

May be, it depends on the typ of connection ... it's a 6Gb/s SATA3 connection ...

Nope. Never ever. Each SATA-III port provides nominal 6Gb/s - after 8b10-encoding: 4,8Gb/s effective (-20%) = 600MB/s.

Samsung delivers an own check-pgm (Screenshot2):

So launch the benchmark test provided by the magician tool.

[Gerd]

... here it is:


[attachment deleted by admin]

[joel23]

... here it is:

Okay, what ever this is, the "Up to" values are more realistic.
How much RAM you have?

EDIT
Can it be that you have Rapid mode enabled? If so I would say that the used RAM is measured. Start it again and stop after some 100MB / 10% or disable Rapid mode to see its real speed.

[cytochrome]

@Gerd:

Seems you have the Magician program. Does it allow an easy switch from a C: drive already on a SSD disk to a new Samsung SSD?

Francis

[Gerd]

Hi,

@Jörg: why shell I disable what delivers the speed? I want to have the max. speed and it doesn't matter, if extra RAM is used or not ... but a SSD works also like a RAM-disk, or ...
I have 16GB installed.

@Francis: what do you mean? My C-drive is a normal hard-disk, only F: is the SSD and the other one's are external's, connected via USB 3.0 (H: and I: have USB 2.0, K: has USB 3.0, my laptop has only USB 3.0)

[sinus]

... here it is:

Gerd, you are about to destroy (torpedieren wäre besser  ) my new knowledge from Ralf, see above and a fraction here, and also Joerg wrote something equal:

For the nowadays most common computer internal interface SATA, there are three different max speeds.

    SATA-6G (or SATA III): max 6Gbit/s or 600MByte/s theoretical transmission speeds (the fastest SSDs reach that value but no harddisk)

    SATA-3G (or SATA II): max 3Gbit/s or 300MByte/s theoretical transmission speeds

    SATA: max 1.5Gbit/s or 150MByte/s theoretical transmission speeds

Such a speed, what you have measured, should simply not be possible ... but of course, I indulge it to you (gönne es Dir).

[RalfC]

Gerd, you are about to destroy (torpedieren wäre besser  ) my new knowledge from Ralf, see above and a fraction here, and also Joerg wrote something equal:

(...)

Markus, don't worry. If you look at the later posts, you will see that Gerd has measured the tranfer rate from a special section of his RAM (internal computer memory) and not the real transmission speed from the SSD.

The "problem" with the speed tests is that they transfer the same data several times and if that fits into a fast buffer/cache, you just measure the speed of the buffer and not the disk itself. I.e., you get to see the real speed only if the data amount does not fit into the buffer.

Regards,
Ralf

[sinus]

Gerd, you are about to destroy (torpedieren wäre besser  ) my new knowledge from Ralf, see above and a fraction here, and also Joerg wrote something equal:

(...)

Markus, don't worry. If you look at the later posts, you will see that Gerd has measured the tranfer rate from a special section of his RAM (internal computer memory) and not the real transmission speed from the SSD.

The "problem" with the speed tests is that they transfer the same data several times and if that fits into a fast buffer/cache, you just measure the speed of the buffer and not the disk itself. I.e., you get to see the real speed only if the data amount does not fit into the buffer.

Regards,
Ralf

Thanks, Ralf, for the clarification! 

And Gerd: you have a good system, no doubt, so you can be happy anyway! 

[Gerd]

... and to let your world turn right  , here the test-data without Rapid-Mode:



[attachment deleted by admin]

[sinus]

Thanks, Gerd! My world turns again from left to right!