Fixing Keyword Issues
Pinned
Here are some notes on what I've discovered after extensively playing with keywords imported from my original Lightroom library. That import introduced several types of errors that have taken me quite a while to clean up. Part of that was my own fault because I embedded those errors without realizing it at the time.
The problems occur when the metadata embedded in the image files and their sidecars get out of sync with the database ON1 maintains. When you make changes to the keywords they are immediately reflected in the database and get saved to the sidecars. If there are any embedded keywords in the image file (you cannot embed with RAW files except .dng) they are now out of sync. If you deleted a keyword or edited one the unchanged version remains in the embedded data and the program will eventually find it again and add it back into the keyword list unless you embed those changes.
ON1 & Adobe use different systems for how nested keywords are stored and identified. Adobe uses pipe characters ( | ) as separators while ON1 uses a left angle bracket ( < ). When I first imported my photos from Lightroom those pipe characters got appended to the ends of the hierarchical keywords which are stored in the metadata. Because I like to keep metadata embedded as well (should I decide to migrate away from ON1 or use another DAM in addition I don't want all the keywords stored only in ON1's database) after cleaning things up I wound up embedding those | characters. When ON1 displays the keywords in the Keyword List it does not show the pipe character. When you double-click a keyword which has been displayed without the pipe character the program cannot find the images it came from. This is because it is searching for "keyword1" when the data it is searching against is "keyword1|". Obviously that is not a match so no results are returned.
Another problem that happened from that 1st import from Lr was that some keywords got a space character prepended to them. Like the appended pipe character this prevents any searches from finding those keywords. " keyword1" ≠ "keyword1".
The trick is to find those images with the problematic keywords and get the database, sidecars, and embedded data synchronized. The way to do that is to create a Catalog for your entire pictures folder. Once that catalog has been built you can select the Pictures folder in the Browser and turn on the option to Show Sub-folder Contents so you can see ALL your photos at once. Do a Select All and you'll see that the keywords which return no search results will be highlighted to show they have been used. Any keyword which is completely unfilled and is not the top level of a hierarchy can be deleted as it is truly unused.
Now if you click the Pictures folder in the Catalogs panel then slide the mouse pointer to the left until it is just outside that panel but still inside the program's window, you can use the arrow keys to walk through the folder structure and expand them to show their sub-folders. As you do this watch those keyword highlights. I worked on one keyword at a time. Walking through my folder hierarchy as I described I could identify the individual folders containing the images with the problem keywords then arrowing through the images in that folder I could find those pictures containing the keywords the program could not find. I did not automatically expand every sub-folder. I only expanded those that showed a hit on the keyword I was working with at the time. Then I would walk down through the hierarchy until I found the individual folder(s) containing the images holding that keyword. No point in walking through my Arizona photos looking for a keyword for a location in Utah, right? ;)
In the Metadata > Keywords panel for the image you can edit the individual keywords and remove any prepended or appended characters that were added improperly. Use the Embed Metadata command to ensure those keywords are no longer in the files for the program to find the next time that folder gets its catalog info updated. Get used to using the hotkeys to expand and collapse the Metadata and Keyword List panels. (Command/Control-8 for Metadata panel & 9 for the Keywords List.) Things go much faster if you're not having to move the mouse pointer from one side of the screen to the other and back again to continue your search.
There is one exception to using the Embed Metadata command. There is a still a bug in how the program handles .dng files. If you use the Embed Metadata command on a .dng file you can watch the Metadata panels Keywords field as the program strips apart the nested keywords and adds them back to the root level with the pipe character appended once more. These are also the files that display a leading comma before the first keyword.
I've been through this process more times than I can count and I've refined my process for identifying the problems and cleaning them up. I'm one of the company's testers so I often completely remove all ON1 from my system and reinstall it from scratch as if it was the first time I've installed the program. This includes deleting the PerfectBrowseCache and and the program's data so the Catalogs also get recreated from scratch. I'm now to the point where the only problems I have with its DAM and keywording is with those .dng files that I cannot re-embed the corrected data to. I've reported all this to the company and I expect that this will get fixed in a future release. They have found and fixed all the other keyword problems I've been able to track down and identify the causes for.
One more note, if you double-click a keyword and get Search Results back, DO NOT MAKE ANY EDITS TO THAT KEYWORD! If you edit the keyword you searched for while viewing the Search Results that image will immediately be removed from the Browser's display and you will not be able to embed its metadata without tracking down again that individual file. It gets removed from the browser display because it no longer matches the search criteria once you've edited the keyword you searched for. If you don't track it down and get the data embedded that image will become a problem again once its parent folder gets its catalog info updated again.
What I did for these cases is to create a "keyword hold" Album. Perform the search, Select All and drag them to the album. Select the album and you are now viewing the same set of images but the keyword search is no longer in force so you can edit the keywords as needed without them disappearing from view as you do so. Once you've made your edits, Embed Metadata, and Remove From Album for that image. After you finish each search's edits make sure to empty the album so it will be ready for the next search.
With this procedure every new release of the program and its attendant rebuilding of the catalogs has resulted in fewer and fewer problems for me until I got to where I am today and the only files I have problems with now are the .dngs with hierarchical keywords. I can live with that now that I can (relatively) easily find and fix them until the engineers get the .dng bug solved.
One other issue I ran across is some times the program cannot write to a file to embed its metadata. The program may or may not report the problem. If you have an image file selected, individually or as part of a group selection, the Embed Metadata command will not be available. I don't know why. Sometimes I've been able to fix the problem by moving that image to a new folder and the command will become available again. Other times that will not work so I use Adobe Bridge (it's free) and work with the metadata there.
If you decide to follow my process, good luck, and ask if you have any questions. It's worth the effort if you want to stay with ON1 and make use of its DAM capabilities.
-
A long story, but a true one. I originally edited and keyworded all my photos (some 20'000) in LR. I chose not to import them into ON1 but to just catalogue them. Of course I lost all my editing, intentionally
But that's when the mess started. Hierarchies falling apart with lower level entries duplicating at the top level, but not searchable. All the issues you describe, I have gone though them multiple time until i found out, that I had to correct each photo's keywords in the individual metadata. But how could I find them when double-clicking the stray keywords wouldn't open the corresponding photo, because they were simply not searchable. Note: half of my LR photos are .DNG, with keywords embedded in the RAW file.
All in all, I spent some 4 weeks on and off editing metadata (some 4000 pictures) and trying to clean up the keyword list. And when my KW list finally looked clean and tidy, It came up messy again after ON1 rescanned all the photos to update its catalog after a restart of ON1.
I had reported numerous issues to ON1 support, I included screenshots and picture files. ON1 was very helpful but couldn't really help, because they didn't have the same basis to begin with.
Thanks Brian, for your extensive report. It doesn't solve problem anymore because I already have done the work. But at least I understand a little better the logic behind what I have done intuitively.2 -
Brian, this is a great write-up. On the DNG files, I'm not sure I'm fully understanding. Are you saying that no matter what cleanup you do, it effectively undoes those changes, leaving the "|" in place as well as the comma in front of the first keyword?
On the deleting keywords and having to make an album as a workaround, is this fix in process? That is quite the problem as well.
Thanks!
0 -
Hi Jerome,
Yes about the .dng files. You can see this happen if you make the changes in the Metadata panel's Keywords field then use the Embed Metadata command. The program will rewrite the .dng file which then causes it to rerender the preview thumbnail. You can see it black out then be replaced by the new (identical) version. Immediately after that the Keywords field will update with the leading comma and the hierarchical keyword lists broken down into their individual components with the pipe character appended. Those updates are added to the .xmp and .on1 sidecars and get added back to the Keyword List without the pipe character.
The process of creating an album to hold search results is not a bug workaround. It's a result of how searching works.
When you do a search the program returns a browser window of Search Results. That is all the images which match the search criteria. If you modify the keyword you searched for on one image, deleting it or correcting a misspelling, etc., that keyword is no longer a part of that image so it no longer matches the search criteria. Therefore it get removed from the Search Results display.
Let's say you do a search for "clouds". One of the photos doesn't have any clouds in it so you delete that keyword from that image. If you're viewing the Search Results that image gets removed because it no longer contains that keyword. If you are trying to clean up your embedded keywords this becomes a problem since the image is no longer available to be selected so you can use the Embed Metadata command.
Moving the Search Results into an album allows you edit the searched for keywords and still see the image to continue working on its metadata.
0 -
Regarding DNGs, so if I 'fix' the keywords but don't 'force' the Embed Metadata command on those files, will the fixed keywords stay as change in On1? If so, I assume they are only in the On1 database for the time being... until a DNG fix is in place and I can safely re-embed the metadata?
I'm ok importing, fixing and then waiting for On1 to provide a 'fix' for DNG metadata, as long as it's on a roadmap within reasonable time, but most of my images are DNG (100K plus) and I really rely on searching metadata.
With regard to both DNG and keyword fixes, is there a global Embed Metadata command to sync all files in a folder, catalog or everything selected? and/or does this function occur on it's own at any point for which I would need to be aware? I would think this could alleviate you deleting/editing a keyword and losing that file back into your collection... being able to get the metadata embedded via a more 'global' process?
0 -
Yes again about not forcing the Embed Metadata command. Things should stay good until that image gets rescanned by the cataloging process which generally happens with every update at a minimum. Changes to the files structure will also cause those folders to be rescanned from what I can tell.
I have no insight into the company's timeline for getting things fixed.
The Embed Metadata command works on the current selection in the browser. If you wanted to force a global update for your entire collection go to the root of your Catalog and turn on the Show Sub-Folders Option at the bottom of the screen. Now do a Select All and then an Embed Metadata.
I do not recommend doing that at this time especially given your extensive use of dngs. Doing that would break down every nested keyword hierarchy in your collection and force you to have to clean it all up again. I do use it on controlled selections of images where I know that the entire set will hold the exact same metadata.
0 -
Ok. Makes sense except for the 'rescan'.
So the rescan that happens 'naturally' will 'fix' or 'sync' the DNGs correctly, then? Trying to understand how it differs from Embed Metadata and/or what triggers this different process, if just writing to that file.
... and thanks for your patience on these details.
0 -
Cataloged folders are monitored for changes to their contents. The OS notifies the program when a monitored folder's contents change. When you embed the metadata that file's update date changes which the program (OS) registers as a change to the contents of that folder so it gets rescanned. When the image gets scanned/rescanned its keywords get reevaluated to see if the keywords database needs to be updated. If there are hierarchical keywords that scanning process is where the bug lies causing them to be broken down, etc.
I'm glad someone has found this useful. :)
0 -
LOL...and THANK YOU. I'm sort of crazy on this stuff.
I've got two Lightroom catalogs... one for my photography business (which I'm finally retiring from) with over 750K images and a personal catalog with just over 200K images that cover scans/pics from 1903 to current... everything in folders by year with year-month-day sub-folders. Very clean and neat.
I keyword everything. In my personal files, I keyword nearly each person, almost always at location, activity, etc. Given the amount of pictures, it would otherwise be impossible to manage... but keyword searching makes it so easy to find everything.
Losing Smart Albums is another story that really has me wondering where the On1 team is going. It's the other big function I use in managing groups of images in Lightroom and a saved search seems a huge step backwards.
Anyway, I'm getting close to cleaning my Lightroom up, backing everything up and readying for a try at the conversion (just of my personal library).
0 -
Before going whole-hog on all 200K images you might want to start with a test of a smaller subset to get a handle on how things work and how it will impact the keywords list. Once you're satisfied that things are working and aren't going to get messed up then do the full library.
0 -
Also, have you seen the Lightroom Migration Tool article in the Knowledge Base?
0 -
Agreed on the 200K, going to take a set of just a few folders with mixed images, export as a test catalog and try small first.
and Yes... I've read that article a couple of times. Doesn't look too bad... seems it just takes a lot of time for a large library.
Appreciate you keeping me honest as well :-)
0 -
Brian, keywords seem to be working for me?!? I'm running 2020.1 (14.1.1.8943), so asking for your review/feedback on what I'm doing.
I cleaned up my Lightroom library today and then selected four folders and exported as a catalog (about 350 images).
I copied these folders to a new location, separate from everything else, and then relocated them in the catalog, followed the migration functions and found them in On1 as expected. This is now a group of isolated images in both Lightroom and On1.
My only goal in this exercise is to test the keyword functionality, as this is the last real stumbling block I have before trying a cutover.
In my browse of images and the keyword list, I haven't seen the prepend of any spaces or commas, nor have I seen the vertical bar "|". In fact, my hierarchical keywords seem to even be ok... nested from top down in the hierarchy with a "<" after each level. I did find a number of hierarchical keywords which were flattened and spaced with commas (e.g., "fabric, cloth, material, ").. this will take some work, but not totally surprised by this finding.
Seeing this looked better than expected, I did the following:
- Played with a couple of DNG files. Changed keywords, embedded the data, retrieved the data and it all stayed as it should. I did this with files that both had existing keywords and some which I imported having no keywords.
- Played with another few DNG files. Changed the keywords, did NOT embed the data, retrieved the data from the fle and it added back in the old keywords which were missing (as expected).
- Did the same with a couple of JPGs, same results.
So, apart from having to ensure I immediately 'embed the metadata' to any files just changed to avoid them being pulled back in to On1 in error, I'm not seeing any issues? It seems On1 is correctly overwriting the embedded keywords .... OR am I not digging deep enough?
Check my process here, or is this 8943 build actually working?
0 -
The appended pipes do not appear in the Keyword List, that is why double-clicking those flattened entries do not return any search results. The only place you can see those characters is in the Metadata > Keywords field for each image that contains them. That is why I described my process for searching for those keywords. I suspect the importer has fixed the problem of prepending spaces. Just in case, double-check the Keywords List. If there are any prepended spaces, those keywords will appear out of sequence at the top of the list. If there aren't any, cool!
What I did to ease finding the flattened versions is double-click on the hierarchical version which should still be in the image metadata. If that search finds all the images with the flattened version too, you'll see that in the Keyword List as it will highlight the circle button and fill in an image count. In this case since you are not searching for the flattened version of the keyword but the hierarchical version you can edit that flattened version in the Keywords field without loosing the image from the Search Results. If it misses some then you'll have to search folders for them. I deleted them from all the images containing them then I deleted the keyword from the Keyword List. If you don't get all the images 1st it will come back.
In regards to item 1. Did any of those test DNG files contain nested keywords? If there are no nested keywords you won't see any problems. If they did have nested keywords and they were not flattened after embedding I can't explain why your version of the program would behave differently than mine.
For item 3 I haven't had any problems with jpegs once I got them cleaned up.
Your process sounds good to me.
0 -
So I did find 4 keywords with spaces right after I posted. Easy enough to fix those, so not an issue. Did NOT find any ',' prepends, but maybe due to smaller set of data?
I did have hierarchical keywords in the DNGs.... here is what ended up showing in the keywords fields for the two I had from a small set of 13 pictures from a work dinner I took a few pics at, hence 'work' and 'restaurant' were hierarchical. There were other keywords such as peoples names, etc as individual items on these files.
On1 shows: ACTIVITIES/ACTION < activity < work, architecture < building < commercial building < restaurant,
Lightroom show: work<activity, restaurant<commercial building
Now, what is interesting is that they are in reverse order from what Lightroom shows and interestingly they show all the way to the top, whereas Lightroom cuts off the parents/below parents (maybe I have to change this in On1?). Also, the '<' symbols seem in reverse of what I would expect as it's reading from left to right as top down, but symbols are inverted? Nonetheless, I still can't find any vertical bars, though I'm suspecting it doesn't happen in all cases and my import set is too small?
What are your thoughts here?
As of now, I think with 200K+ images , this effort is otherwise going to be too long and leaning toward completely documenting my hierarchy, flattening it in Lightroom before exporting and then rebuilding in a more clean/simplistic way in On1. This might avoid the problem entirely, take less time and provide me a keyword house cleaning opportunity.
What's occurs to me is that On1 does a pretty good job of getting the data across, apart from the nesting issues. Takes a little cleanup work, but if flattened, it should be really easy.
On the "re-appearing metadata", On1 should really add a preferences checkbox or functionality to turn off/on that it won't RE-READ metadata from a file (at least in a catalog folder) unless you force/ask it to (and should default to such unless you over-ride). This would entirely avoid the problem of these overwrites after the fact and make it a one way street once you start working in On1. It's a simple 'exclude' functionality to their scan process to ignore metadata from the files if rescanning, if such a box is checked in Preferences.
Feel like I'm making some progress and getting more comfortable, but for now the most mission critical lesson is that if you change metadata, you had better 'embed' as you go or you'll pay the price later.
Thank you for all the continued feedback. It's great to have someone to converse with on this challenge.
0 -
Do we know if On1 Mobile (360) is going to allow keywording? Will it be only flat keywording, as we have with Lightroom CC (mobile version)?
... should I ask this elsewhere to get the On1 teams' attention?
0 -
Jerome,
The comma prepends are just a visual artifact in the Keywords field. I don't think they are a representation of any data problems so I don't worry about them.
The order in which the keywords are listed are a design preference. Same with the choice of the < separator to denote the hierarchy.
Where are you looking for the pipe characters? They only show up in the Metadata > Keywords field for DNGs after I've used the Embed Metadata command. If you're not seeing any of them that's a good thing. You're ahead of the game. :) I just rebuilt my Catalog again last night and I haven't seen any more pipes either. I'm hopeful I've finally eliminated them. It is possible that it only happens with certain files. If so, I haven't yet found a cause for why it would happen with one file but not another.
I don't have an opinion on flattening the keyword hierarchy prior to moving the photo to ON1. You can see what you're working and I can't so I'll rely on your expertise here. :)
For the keywords reappearing issue, it wasn't a problem for me with the catalog rebuild I did yesterday.
As I recall, Lightroom would automatically embed metadata changes. That could be a preference setting I chose. It's been so long since I've run the program I've forgotten.
0 -
Jerome,
The company hasn't released much info about the mobile program. All we know is what we've seen in the introduction videos Dylan's been producing lately. I can't image it having only part of the keywording features though. That would make it very tough to keep things in sync.
0 -
Brian,
Definitely looking for those pipes in the Metadata > Keywords field. I'm guessing there is some scenario which causes it and my dataset is too small. I guess I'll just have to roll some dice on that one. At least there is hope to getting in cleaned and stable.
My idea on flattening is a "Pay me now or pay me later" kind of thing... and if I want to clean the hierarchy it seems it would be a good time to tear down and avoid the 'pipe gamble' all together. Will have to think on that one.
I think the lack of the 'automatic embed' in On1 along with whatever scanning it is doing creates the problem folks are seeing. If I embed, as you did, it should synchronize and avoid the issue.
On the mobile app, Lightroom is not consistent with their approach. They allow hierarchical keywords in Lightroom Classic CC and ONLY flat keywords in Lightroom CC (mobile). Hence, if On1 is going this route as well, a flat hierarchy might be best overall and why I'd like to know before I start down this road.
0 -
I think you've got a solid plan Jerome. You'll have to ask support about the keyword features expected in Mobile.
0 -
As far as I can tell the DNG issue has been resolved. I'm no longer seeing hierarchical keyword structures being stripped apart with pipe characters appended to their components.
0 -
Excellent! Very excited about this and thank you for the update.
However, I'm now at a bigger crossroad... facing a much different issue which is the mobile app is completely unusable.
Crashes on importing from SD card, external SSD hard drive (both of which work with Adobe just fine and my phone file explorer).
Camera app crashes as well... won't work at all.
Then, in both cases, the app won't reopen, most of the time. Have had to reinstall too many times. If this mobile app is useless, then I'm staying with Adobe.
So, for the time being, my cutover plan is completely on hold.
0 -
Brian, I just tried 'Find cataloged photos with keyword' but it only searched Current Photos, not catalogs. Are you seeing that?
I can manually select catalogs, but that should be set automatically by the search.
In fact, if I do set it manually and then re-try the search, it changes back to current photos.
0 -
All new searches now default to searching Current Photos. I think that was done in response to the delays that happened when typing in a search parameter while searching the entire catalog. I just change the search location in Advanced Search to search the entire catalog.
0 -
That means the wording should be changed in the menu because if you don't notice it, you'll think there are no matches.
0 -
It would probably be better if searching through the Keyword List, whether double-clicking an entry or right-clicking and choosing the search command, defaulted to the entire catalog while searches typed into the search fields defaulted to the current folder. Best of both worlds IMO.
0 -
You're right, the command explicitly says Search Cataloged Folders. I usually just double-click the one I want so I hadn't noticed that. That looks like a bug to me.
0 -
Wow, that is annoying. You have to look in both locations to make sure you're searching right. Wouldn't it just be easier to make the Cancel button work?
Double clicking works the same now. Only current photos.
0 -
Yes, that I had noticed. I'm going to report it right now.
0 -
Brian,
I have already opened a ticket for that #239598 to make the default cataloged photos. They acknowledge it is a bug and are now working on it.
0 -
Cool, thanks, Gerry.
0
Please sign in to leave a comment.
Comments
40 comments