OCR with Gemma 3

[Mario]

Did you know that Gemma 3 is quite good at extracting text from images?
I have not compared it with what cloud-based models can do, but for my limited tests, the results were very good.

I used this simple prompt with Gemma 3 12B running in LM Studio:

Extract the text in this image and return it in Markdown format.

And this was the result (PDF in QuickView Panel, resulting XMP description in Markdown in Metadata Panel, looking at the description in a Markdown editor (open the image in a new tab for a larger view):

[bekesizl]

I am interested in this feature as my other tool, Mylio is using the built-in OCR feature in Windows to analyze the imported photos.
It is a really cool feature to be able to search for text and find pictures having text on them without categorizing them first.

I made a test with 3 images containing text in Hungarian or German language.
The results were really bad. I attached one example to this post.

I used following prompt for Gemma 3 12B (with LM Studio):

Extract the text in this image and return it in Markdown format. 
Only respond with text actually found on image.Stick to the facts.

At first with normal image size, then with the maximum.
I started with the original prompt from Mario, then extended it to the above form.

The response from AI looked like that it could read the title of this table and made up the rest of its content.
The original text is about the Victoria Lake in Africa. The AI response was about a Victoria Lake in Hungary.

My guess is, that the rest of the text got unreadable when the image arrived at the AI.

It is clear to me, that the text is not as clear readable as on the document of Mario. Contrast is less, text is distorted because of the camera angle. But it is good readable at full size.

I didn't have the chance yet to send it to a cloud AI like Mistral.

I guess that it might still be needed to handle images for OCR differently, like sending it in full size to the AI (if it can handle it).

Maybe this is also a bad example.

[bekesizl]

This is another example. The attachment is reduced to 80% size in Irfanview (original file size was bigger than the maximum allowed attachment size).
When I fed this image to Gemma, it started to write to me about some power outage in Tirol, Austria.
Again it was made up from the title of the image.

[Mario]

When you have Mylio, why use IMatch?

If Mylio does better face recognition as per your email and the dedicated OCR in Windows (which I've got very mixed results from compared to real OCR software) also works better for your files than Gemma, Gemini or OpenAI, why not just stick to Mylio?

Windows OCR in OneNote or similar tools is dedicated to the job of OCR. For Gemma, this is merely a side show. Results will vary. If you are in need of OCR regularly, use Windows OCR or a dedicated OCR tool.
Looking at zero feature requests for OCR in IMatch, I don't thin many users will ever have a need for it.
Feel free to add a feature request in the feature request board. Other users can then comment on and like the request.

If there are requests, I would probably integrate Windows OCR, when it is royalty free to use (I don't know).

[Mario]

Feeding the two images into Gemini with large image size produces:

## Mit ohne Strom

Leben wie damals

### Wasserkraft

Es gehört zu den elementaren Kulturleistungen des Menschen, die im fließenden Wasser enthaltene Energie zu nutzen, um sich Muskelkraft zu sparen.

Mühlen zeigen verschiedene Antriebstechniken, vom direkten Antrieb des Mühlsteins bis zu den oberschlächtigen Wasserrädern. Bei diesen fällt das Wasser von oben in einzelne Fächer des Rades, belastet es bis zum tiefsten Punkt und versetzt es dadurch in Bewegung. Intensiv wurde die Wasserkraft in Hammerwerken bei der Metallverarbeitung oder Sägewerken genutzt. Welche Technik zur Anwendung kam, hing von den örtlichen Bedingungen ab. Für die Effizienz war die vorhandene Wassermenge und das Gefälle verantwortlich.

Richtungsweisend für die optimale technische Nutzung von Wasserkraft war der Bergbau. 1556 ging in Schwaz in Tirol eine Anlage in Betrieb, die vier große Pochwerke antrieb und sowohl Erz als auch Grubenwasser aus der Tiefe beförderte. Mit der Einführung der Dampfmaschine um 1780 kam eine weitere Kraftquelle hinzu.

Im bäuerlichen Umfeld hatte die Wasserkraft noch bis ins 20. Jahrhundert eine zentrale Bedeutung. Die Anlagen wurden technisch immer weiterentwickelt. Heute produziert Wasserkraft primär Strom. Die Ortsgebundenheit der technischen Anlagen ist dadurch nicht mehr notwendig - außer es gibt Pannen, dann steht weiträumig alles still.



## Viktória Tó

**Története**

Az első feljegyzett írásos emlék a Viktória-tóról arab kereskedőktől származik. Abu Abdallah al-Idriszi, arab térképész 1160-ban készített térképén pontosan ábrázolta a Viktória-tavat és fontosabb jellemzőit. A tó első európai felfedezője John Hanning Speke volt, aki 1858-ban érte el a tó déli partját, tőle származik az elnevezés is Viktória királynő után. A híres brit felfedező és misszionárius, David Livingstone kísérlete meg igazolni Speke felfedezését, mivel abban az időben nem tekintették elfogadottnak, hogy a tó valóban a Nílus forrása lenne. Később hasonló útra vállalkozott a walesi-amerikai felfedező, Henry Morton Stanley, aki végül megerősítette Speke felfedezésének igazságát.

**Földrajz**

A Viktória-tó az afrikai kontinens legnagyobb tava, a világ legnagyobb trópusi tava és felülete alapján a világ második legnagyobb édesvízű tava. A méretéhez képest sekély, átlagosan 40 és legfeljebb 84 méter vízmélységű, ezért vízmennyisége alapján (2750 köbkilométer) csak a hetedik, a világ édesvízű tavai közt. A Viktória-tó a Nílus leghosszabb felső ágának, a Fehér-Nílusnak a forrása. Vízgyűjtő területe 184 ezer km². A tó vízének 80%-át közvetlenül a területére hulló csapadékból kapja (melynek mértéke 1015 mm évente), illetve több ezer kis patak, valamint a Kagera-folyó táplálja. Nagyjából téglalap alakú, 320 km hosszú (észak-déli irányban) és 275 km széles (kelet-nyugati irányban), számos félsziget, öböl, és több, mint 3000, többnyire lakatlan sziget tartozik területéhez. A Viktória-tó vízszintjének alakulása a Viktoria-tó nagyon fiatal, becslések szerint egy millió évnél is fiatalabb keletkezésű lehet. Az utolsó periódus, melyben a tó teljes területe szárazulat volt, 14 700 évvel ezelőtt volt. A Viktória-tó egy fennsíkon fekszik a Nagy-hasadékvölgy mentén, területén Tanzánia, Uganda és Kenya osztozik. Partvonala 3440 kilométer hosszú, több mint 3000 szigete van, sok közülük lakott. Ezek közt van az ugandai Ssese-szigetcsoport a tó északnyugati részén, amely népszerű turistacélpont.

**Élővilág Állatvilág**

Területén élő emlősök közé tartozik az antilop, az impala, a víziló, a vidra. Jellemző madárfajai a mocsári légykapó (Muscicapa aquatica), a fehér szárnyú poszáta (Bradypterus carpalis), a kárókatona, különböző gémek, a nemes kócsag, a sirály, a csér (Sterna nilotica) és a jégmadár. A Viktória-tóban 550 különböző halfaj él. Jellemző faja a sügér (Cichlidae), egyes becslések szerint akár 400 különböző faja is élhet a tóban, ebből eddig 125 fajt irtak le tudományosan. Jellemző rovarfajai közé tartoznak a különböző kérészek, szitakötők, szúnyogok. Sok rovar betegségeket is terjeszthet, mint például a malária és sárgaláz, illetve a cecelégy által terjesztett álomkór. A rákfélék is igen elterjedtek a tóban, főbb képviselőik a garnéla (Caridina nilotica) és a tarisznyarák (Potamonautes niloticus). 126 kagylófaj él a tóban, valamint elterjedt a pióca és az édesvízi medúza is.

Which looks OK to me. As always, specialized skills like OCR are best provided by specialized models, as OCR software and Windows OCR (presumably) use.

I can, in the same prompt, ask for a translation into English, too. Which is nice:

## Lake Victoria


**History**
The first written record of Lake Victoria comes from Arab merchants.
Abu Abdallah al-Idriszi, an Arab cartographer, depicted Lake Victoria and its more important features accurately on a map made in 1160.
John Hanning Speke was the first European explorer of the lake, reaching its southern shore in 1858. The name comes from Queen Victoria.
British explorer and missionary David Livingstone tried to prove Speke's discovery, as it was not accepted at the time that the lake was indeed the source of the Nile.
Later, Welsh-American explorer Henry Morton Stanley undertook a similar journey, and ultimately confirmed the accuracy of Speke's discovery.

**Geography**
Lake Victoria is the largest lake in the African continent, the largest tropical lake in the world, and the second largest freshwater lake in the world by surface area.
It is shallow for its size, with an average depth of 40 meters and a maximum depth of 84 meters, so it is only the seventh largest lake in the world by water volume (2750 cubic kilometers).
Lake Victoria is the source of the White Nile, the longest upper branch of the Nile.
The catchment area of the lake is 184,000 km².
It receives 80% of its water from rainfall directly on its surface (with an average of 1015 mm per year), as well as thousands of small streams and the Kagera River.
It is roughly rectangular, 320 km long (north-south) and 275 km wide (east-west), and has numerous peninsulas, bays, and more than 3,000 mostly uninhabited islands.
The formation of Lake Victoria is very recent, with estimates suggesting it may be less than a million years old.
The last period when the entire area of the lake was dry was 14,700 years ago. Lake Victoria is located on a plateau along the Great Rift Valley, and Tanzania, Uganda, and Kenya share its territory.
It has a shoreline of 3440 kilometers, and more than 3000 islands, many of which are inhabited.
Among these is the Ssese Islands in Uganda, in the northwestern part of the lake, which is a popular tourist destination.

**Fauna and Flora**
Among the mammals living in the area are antelopes, impalas, hippos, and otters.
Typical bird species include the swamp flycatcher (Muscicapa aquatica), the white-winged warbler (Bradypterus carpalis), the cormorant, various herons, the great egret, the gull, the tern (Sterna nilotica), and the kingfisher. There are 550 different species of fish in Lake Victoria. The cichlid is a characteristic species, and some estimates suggest that as many as 400 different species may live in the lake, of which 125 have been scientifically described. The common insect species include various mayflies, dragonflies, and mosquitoes.
Many insects can also spread diseases, such as malaria and yellow fever, as well as sleeping sickness spread by the tsetse fly.
Crustaceans are also very common in the lake, with the main representatives being the shrimp (Caridina nilotica) and the freshwater crab (Potamonautes niloticus). There are 126 species of shellfish in the lake, and leeches and freshwater jellyfish are also widespread.

[bekesizl]

When you have Mylio, why use IMatch?

The only essential point for continuing using Mylio is its ability synchronize and make available all the photos in my collection to different devices, like my phone.
And this in a privacy conform way.
And in a searchable way. At least for metadata fields relevant for me.
It is something I haven't seen in any other tool.

But this synchronization can cause that much trouble (at least for a complex system) that was multiplicating a lot of my files from the last years, making unwanted copies of 0,5TB to my 1,2TB collection.

That takes me to use IMatch for my original files and create a shadow library to be seen by Mylio, so that I can have my photos available.

As on the other hand, IMatch is doing an excellent job at managing my files and is also of great help to clean up the mess the synchronization issue caused.

I am not an IT expert to try to make my IMatch available online on my phone in a safe way.

If Mylio does better face recognition as per your email and the dedicated OCR in Windows (which I've got very mixed results from compared to real OCR software) also works better for your files than Gemma, Gemini or OpenAI, why not just stick to Mylio?

OCR is nice to have. But as the AI models evolve, there might come some with great OCR capabilities and if their interface is compatible to the ones you are using now, it might be useful to integrate them.
But it is a really cool feature to be able to find a picture with a text on it without doing to much for this.

Face recognition in IMatch is good. The tagging in Mylio is better in the way, that more faces get recognized. There was a suggestion some weeks ago here in the community forum to make the recognition in multiple steps to have more faces recognized.

The problem in this area with Mylio is, that it starts to reanalyze all your files if a new version arrives. Or if you have to start your library again after a big synchronization issue. And then you have to ignore all the faces you ignored previously.
This would also be the case with IMatch if I remove files from the database and add them again.
But with IMatch I take out the complexity of the synchronizations, so the probability of such a failure gets much much less.

If there are requests, I would probably integrate Windows OCR, when it is royalty free to use (I don't know).

I spent some time in searching for this and came to the conclusion, that it is not worth the trouble. There was something I found on Github, but I can't find it in the moment.
Integrating an AI model like mentioned above might be easier.

Feeding the two images into Gemini with large image size produces:

Thank you very much for taking a look at this.

This result looks much more promising. The text is OK.
If you can make AutoTagger able to send a larger image to Gemini, the whole thing might be solved.
And when I think of how you work and how well you prepare your code, this might not be a big work for you.
And I can make a special configuration for OCR with Gemini and only send it the large size image for this purpose.
Or maybe Mistral also creates something similar and then I will have even less trouble to send there a picture containing a face and a text.

I am using Gemma to create a Trait if the image has text on it. The 4B model was not very good at this, but the 12B is good at this job.
This way I can filter out images that I would need to send to Gemini for a more in depth analysis.

[Mario]

If you can make AutoTagger able to send a larger image to Gemini, the whole thing might be solved.

You can select between normal, larger, extra-large and thumbnail directly in the AutoTagger settings: Image Size

For small text I recommend to use large and when this does not work, extra large. Larger images take longer to process and cost more. For example, Gemini takes 384x384 pixels as the "normal" image size, which works well for most images. For text or detecting small details, you can create a "Large image" setting and use it when needed. AutoTagger has you covered.

I am using Gemma to create a Trait if the image has text on it. The 4B model was not very good at this, but the 12B is good at this job. This way I can filter out images that I would need to send to Gemini for a more in depth analysis.

Very good. At least at this time, model size is important. Every time a model is condensed (Gemma 3 27B, Gemma 3 12B, Gemma 3 4b) some precision and capabilities are lost. If we could run the 27B Gemma 3 model (which requires about 30Gb of VRAM), it would beat the 12B model.

The ~400B to ~500B models OpenAI, Google etc. run in their clouds of course perform a lot better. Luckily, Gemma 3  (12B) works extremely well for the task at hand: creating "good enough" keywords and descriptions. I'm very happy with the results I get from Gemma 12B.

As you said, for "more special thing", we can always resort to OpenAI, Mistral or Gemini in the cloud. Just by switching to another setting in AutoTagger.

[jch2103]

Without going into a lot of details, I used the same LN Studio prompt (nothing specific about text recognition) from https://www.photools.com/community/index.php/topic,15076.msg105707.html#new on the attached image.
The AI description included "Buffalo Bill's Resort and Casino in Primm, Nevada." Not bad, considering that the image was a smartphone shot taken by a relative from a moving car!

[Mario]

Looks good. Note that for OCR / text recognition, your setting should specify the "large" image size. Most AIs use 384 or 512px for the normal image size, which can be too small for OCR or small details.