Skip to main content

Recovering from Lost Edits

Comments

23 comments

  • Brian Lawson Community moderator

    Jerry,

    I would be sure to keep last night's backup until all this gets resolved. Some users have found that the Refresh Preview command under the Photo menu will reconnect the image with its edits.

    With the CPU/GPU usage you're seeing I'm betting the cataloging process is still taking place. Let it finish. If it doesn't help you won't loose anything as long as you keep that backup. Go to the My Catalogs tab in the Browser and hover over each of the Catalogs you've created to see their progress.

    No, I don't think reformatting your SSD will be necessary. Do be sure to let support know about all of this and follow up tomorrow here and with them on the progress after the catalog finishes.

    0
  • Jerry Hall

    Thanks Brian.  

    I have sent Support all this sad history.  I am afraid I have overwhelmed them with info.

    All the catalogued folders show 100% (neat suggestion) but GPU and CPU are still at medium high levels.  Will restart tomorrow.

    Tried Refresh Preview on one image, no luck.

    When this is resolved I am going to stay away from the catalog.  I will read up on what I will loose.

    0
  • Jerry Hall

     

    Current situation (cc'd to support):

    ON1 will not recognize nearly all of my edits on my internal drive.   Exception is my Google Drive where I saved my print images and their source.  

    No edits are recognized on my external SSD.  The +/- badge does not show.  

    My nightly back up for the external SSD on a hard drive is a mixed story.  It seems starting a few weeks ago images that were adjusted and then backed up migrated into a state where ON1 does not recognize the associated .xmp and .on1 files.  

    These sidecar files remain in every case on internal drive, external SSD and hard drive as far as I can see.  So why can't ON1 recognize them in browse mode and feed them to Develop?  Finder sees them.

    I am now dead in the water for new work, and little confidence in how to proceed.  I am becoming reconciled that I will have to re-\\edit most of my images and salvage what I can.  I have run a small batch test verifying that I can move badged files from my hard drive backup to a new folder on my external SSD

    My tentative recovery plan is:

    1. Uncatalogue my external SSD (I am done with the catalog for now)
    2. Establish a parallel folder structure on external SSD
    3. Migrate folder by folder from backup hard drive and check results as I go along. 
    4. Backup my best raw keepers and print files on google drive
    5. If any problems occur start trials on alternative software

    I've put all my eggs in the ON1 basket.  It is a well designed product.  Of course software that can't retain access to your data isn't worth much.  I hope this is a passing problem, but without understanding how it came to be I will always cast a wary eye.

    0
  • Jerry Hall

    Brian,

    Support says "Making any changes (renaming/moving)  to the folders that were cataloged would cause issues, removing the catalogs and also resetting files.moving, renaming catalogued folders can cause problems."  This is not how Browse is described in the manual p30 which advertises a full finder like capability for moving etc.

    Support suggests trying going back to the 2020.0 version as that has helped in some similar cases to make "lost" edits visible in Develop.  What do you think?  Risk of making things worse? I am not sure about the process, eg retain old version etc.

    A disappointment is that my external backup hard drive (never catalogued) shows only a few +/- badges (the edits show up in Develop).  Sidecars are there for all files per finder.  I think the timing of the last incremental backup infected some but not all the files.  I am beginning to suspect that the .On1 files that were backed up from my problem catalogued folders have been encoded somehow to prevent them from being read by Develop.

    If the suggested install of the old release doesn't work how can I get a fresh start from my hard drive backup files with a new install.  Stay away from the catalog altogether?  Is there any hope of my sidecar files being of use or should I just keep my raw files and edit anew?

    Future considerations: How do I backup? What if I want to upgrade my external drives, or get a new computer.  I don't have a clue how that should be safely done.

    It's a mess.  Appreciate your advice.  Wonder how a more complex ON1 360 Browse will work.  

    Jerry

     

     

     

     

    0
  • Brian Lawson Community moderator

    Wow, there's a lot in there. :)

    This is not how Browse is described in the manual… I agree, that's why I've suggested moving things as a possible fix. However, because of the database issues right now, I'd be inclined to go along with their recommendation.

    Support suggests trying going back to the 2020.0 version… I've read here and on Facebook from users who've said rolling back to an earlier version solved their problems. I do not think it will make things worse. What you have to decide is if the database issues are more important than the other bug fixes and program improvements that come with the updates.

    Rolling back isn't hard, you just uninstall the current version then run the installer for the older version. I keep backup copies for all releases for just this reason. I don't use the Upgrade offer from the program even though you get the exact same updater either way. When the program does it the updater is hidden in a temporary location making it hard to find and save. It's easier to just download the latest installer from the web site. If you don't have the previous installer ask support for one and they'll give you a download link.

    A disappointment is that my external backup hard drive (never catalogued) shows only a few +/- badges… Your last backup would have backed up any of the .on1 files that had been updated since the previous backup but that's what it's supposed to do, right? ;) You mentioned the sidecars being "infected" or "encoded". I don't think this is happening. The files are just an .xml and easily read by any text editor. You can open one up and look to see if the editing information is still in them or not. A lot of the entries are just the slider control name followed by its setting value so it is pretty straightforward to identify what's going on without having to understand it all and to identify if the editing info is missing or just not being recognized by the program. You could also compare one from the backup still showing its ± badge with one from your working drive that is missing the badge to see if their contents differ. When I've looked at some of mine I've always found the edit data there.

    [H]ow can I get a fresh start from my… backup? If the editing data is all intact reinstalling the previous version should pick it up as the new (re)installation rebuilds its databases. I would think the same thing would happen if the program was completely removed from your system and installed from scratch as if it was your first time installing it. That does rely on the editing data being intact in the sidecar files though.

    If the problem is that the edit data is in the database and not the sidecars but the program is letting the sidecar override the database so the edits don't show then an uninstall will cause the edits to be lost permanently.

    If you have an incremental backup from prior to the update you can restore to that point and loose any edits you've made since the update.

    Your other option as I see it is to hang in there and see what the engineers come up with to fix this. It may or may not include recovery of the "missing" data. Hopefully it will. That's all going to depend upon where it is missing from and I don't have a complete handle on that. I've tried looking at the databases but they are encrypted.

    Future considerations: I've migrated my photos from 4 different machines now and from desktops to laptops. Early on I moved my photos to their own internal drive on my desktop system. This was for performance reasons as well as making it easy to move and back up the photos separately from the rest of the system. Over the years they have also been moved from one drive to another and from one technology to another. The transition has always been smooth. All I've had to do is plug in the drive and I'm back in business whether it is restoring a lost drive or moving to a new machine.

    I have multiple backups of the photos drive. (All my drives actually.) I keep one of them at another location for disaster recovery. Every once in a while I'll bring it over and up to date then take it back. Because of these problems with edits going missing I recently added a Time Machine drive for incremental backups so the next time I find an edit has gone missing and the .on1 sidecar has changed I'll hopefully be able to backtrack through the incrementals to find the point at which it was changed and recover the lost data.

    I use Carbon Copy Cloner to make my nightly backups which are mirrors of the drives they backup. I can boot from my internal boot drive backup if necessary. It uses an incremental backup strategy too but it isn't as long term as the Time Machine drive. I also archived my Mojave OS with it before upgrading to Catalina so I can go back to that and my CS6 copy of Ps if needed. All of that is on a single external 2TB SSD — Mojave OS segregated into its own APFS container; Catalina OS  and my photos drive each on their own partition in a shared APFS container. When I plug in the drive it shows up on my desktop as 3 separate drives.

    I hope this helps.

    0
  • Jerry Hall

    Brian,

    What a responsive reply.  Lots to think about.  Thank you.  You are a real asset to this forum.

    I went back to 20.0 as suggested.  Many edits have become accessible, but many are still not.  I waited to report since there was a lot of GPU and CPU activity and there was evidence that progress was being made.  Think it has peaked tho.

    Are you suggesting that the edit history in some circumstances is not just logged in the .on1 files but in some mystery encrypted database?  That would explain why having the two sidecar files sometimes is not enough to restart and continue adjustments in Develop.  If so, is it also true for folders that have never been catalogued, or in that case is the edit history current in the sidecar?  If the latter that is a very good reason for me to not catalog.  I can live without the benefits of the catalog feature if my DAM is rock solid.

    BTW my backup approach is very similar to yours with CCC.  But I am shook by the possibility that having sidecar files in a backup may not be good enough for ON1.

    Do you think ON1 is rock solid as a DAM if I stay away from catalogued folders?  If so I will take my lumps and head that way.

    If not then do you know of a third party browser that will work somewhat effectively in single image mode?  Need be I can use my careful folder structure and Finder and Single Image Edit.  I do not want to go to another editor.  ON1 is great in that regard.

    Thanks for hanging in there with me.  My work experience has taught me not to bet on the come with promised fixes, but to stay on a solid foundation before upgrading.  I need to heed that lesson.  I am trying to figure out what that foundation is so I can get back to creating images.

    Without your help I may well have given up and returned to my woodworking shop!  I wonder if ON1 has lost the institutional knowledge of the original Browser engineers.  Stuff like this has been going on for a while and it smacks of staff turnover.  Fortunately I am a new photographer and it won't kill me to reedit some pictures for print with my now improved skills.  I just want to get my DAM feet under me and not worry about losing stuff.

    Jerry

    0
  • Brian Lawson Community moderator

    Are you suggesting that the edit history in some circumstances is not just logged in the .on1 files but in some mystery encrypted database? No, I'm not suggesting it, I'm outright stating it. :) Sidecar files are optional. They can be turned off in the program's Preferences. Without them the edits must be stored somewhere or be lost, right? I see sidecars as a backup to the database.

    Cataloging and the saved edits should be independent things. Cataloging is what turns the program from a simple browser into a DAM. It is what gives you the ability to search your library for specific images without having to know where they reside before you start.

    From what I understand the problems with the lost edits have something to do with how the program handles external drives but I don't know any details. Cataloging is tied into that of course since it searches and monitors changes to those drives.

    I'm comfortable with ON1's DAM capabilities. I can certainly understand where a professional photographer cannot afford to loose edited images, it certainly sucks enough as a hobbyist but I'm not loosing potential income if I have to redo something. Adobe makes Bridge available for free. I occasionally use it in conjunction with ON1 especially when I was troubleshooting the problems I was having with keywords and files that ON1 couldn't write to when I used the Embed Metadata command.

    0
  • Jerry Hall

    That clarifies things a lot.  Should have remembered the .on1 files are a preference.

    Catalogued folders with sidecars invoked are clearly not aways a backup in my case as in many cases they are not recognized.  That is what you meant when you said: "If the problem is that the edit data is in the database and not the sidecars but the program is letting the sidecar override the database so the edits don't show then an uninstall will cause the edits to be lost permanently."

    Again, at this point would I be safer, in an abundance of caution, to avoid catalogued folders because there may be some ON1 behavior that disables the .on1 file as a backup?

    If so is there a safe way to un-catalog?  I assume copy/save all files to an uncatalogued folder system.  That will save what edits there are.  Then un-catalog or delete the old folder system?

    Good suggestion on Bridge.  Will check it out.  Hope that avoiding catalogs and staying with  Browse will do the trick.

     

    0
  • Brian Lawson Community moderator

    Catalogs are stored in a separate database from the edits. You can safely turn them off by right-clicking on a Cataloged folder and choosing the Remove Cataloged Folder command. It does not change your files or anything about the editing data, it just removes the catalog database entry for that folder.

    Catalogued folders with sidecars invoked are clearly not aways a backup… Right now that is true but I don't think it has anything to do with the Catalog. I could be wrong though, that's just my opinion. I think even without catalogs or sidecars the edits can still get lost, especially on an external drive.

    If you are more comfortable not using catalogs for now, go for it. On one of the other threads here today someone posted a message they'd received from support that said the .5 update will be out at the end of June.

    0
  • Jerry Hall

    I think even without catalogs or sidecars the edits can still get lost, especially on an external drive.

    From Support: We are investigating this issue, but unfortunately have not been able to reproduce it.

    Not a confidence builder for .5 release.

    Just for grins tried copying one folder from SSD and from backup hard drive back to its prior location on internal drive.  No difference I can see,

    Just have to assume that edit info is not guaranteed for now.  Guess I will plunk along for now as I am, with catalogs, and be sure to save my finish edit print images as .tif and/or full size jpegs.  Between that and my raw files count myself as backed up.  Reassess ON1 after .5 release.  

    Not good at living with ambiguity.  

     

    0
  • Jerry Hall

    I noticed today that my external SSD no longer shows as Catalogued and all edit badges are gone.  To the best of my knowledge I did not un-catalog those folders.  Implies to me that once the catalog info was gone all edits are gone and that ON1 was not routinely updating the .on1 files on external SSD.  

    The few files I have edited on catalogued folders on my internal SSD have retained badge and edit info.  It seems there is a problem handling external drives.
     
    Also ALL my key word linking is now gone.  The list is there and complete but linkage to individual files is all gone.
     
    I am really stalled in my confidence to use ON1.  Everyone has done their best and have been very responsive, but the lost hours of edits and work in progress hurts.
    0
  • Rick Sammartino Community moderator

    Sometimes if you start up On1 and a catalog is missing, you can just close and restart and the catalog could come back.

    As long as you have sidecars, all of your edits should be there. Photos do not need to be cataloged at all, that's just for searching your library.

    Sounds like something broke the link between On1 and your external drive.

    0
  • Jerry Hall

    Rick, Brian,

    Current status: External SSD has lost all its edit badges, no history shows up in Browse Info tab,  all sidecars are present, they just aren't read by Develop.  All keyword linkages have been lost.  Restart makes no difference. Am on 2020 release.

    Support has sorta endorsed my following plan going forward to isolate myself somewhat from the external SSD and cataloguing:

    • Un-catalog all internal folders. Leave external SSD un-catalogued as it is now.
    • Set up folders parallel to my SSD on my internal drive google drive folder for redos of my seriously edited images. (I have had no loss of edits from this drive/folder). 
    • Use my SSD as an import repository and add new folders for new shoots. Basically keep it as a repository for digital negatives. Use custom presets on SSD for preliminary evaluation of keepers. 
    • Copy files for serious editing to google drive.
    • Wait for results of June release to see if external drives are handled better. 
    • Watch to see if I lose sacrificial SSD presets/edits.
    • Do not use keyword tagging.

    Would appreciate your comments on this plan to buy some time and maybe help diagnose the problem.

    It is clear there no reason to believe the June release will better handle external drive problems and we are left with ongoing uncertainty.

    No doubt Support is concerned about the multiple reports but can't duplicate the problem.  I am concerned that this is a back seat issue with the new 360 offering.  It leaves me with a poor set of choices going forward.

     

    0
  • Brian Lawson Community moderator

    Your plan seems sound to me. I appreciate you letting us know support's response.

    Maybe I'm just hopeful, but I'm pretty certain the engineers are working just as hard on this as they are on 360º. Loosing edits is a killer for an editing app. If it isn't fixed in the June release I'd bet some users will start leaving.

    Sometime this weekend I'm going to do a complete uninstall of the software and see what happens there.

    0
  • Rick Sammartino

    Yes, appreciate the update, sorry for all the troubles.

    I don't understand why edits would be missing when sidecars exist. Preventing that is the whole point of sidecars.

    It would be interesting to know if On1 is just not reading the sidecar, or if a blank edit was accidentally written to it at some point. If you are unable to check that, and you want to make a photo and it's sidecar available, we could check that for you.

     

    0
  • Brian Lawson Community moderator

    The .on1 sidecar files are just an xml format text file and can be read by any text editor. You'll see the names of the various sliders and their settings if the edits are still intact in the .on1 file. You'll also be able to find the names of the various filters. You don't have to understand the format to see if the edits are there or not — just look for the names.

    0
  • Jerry Hall

    Brian, Rick,  Here are two examples of the info I think you wanted to see.  Interesting one of the key words did link up.  I am going to look some more but I think a lot of my .on1 files have disappeared from the SSD from files I am sure I edited because I had color flagged them. Thanks for your persistence on this.  Jerry

    0
  • Brian Lawson Community moderator

    Those particular files do have the metadata but no editing data. Bummer. 

    0
  • Jerry Hall

    Brian,

    Don't know if this will help but a while back I happened to copy an edit on the SSD to a catalogued internal drive print folder.  Comparing the two the SSD file has no edits but an .on1 file.  The internal Drive file has edits, presumably from the catalog, but no .on1 file even tho I have always used the side car option in preferences.

    Anything to be made of this?

    Jerry

    0
  • Brian Lawson Community moderator

    I don't know what to make of that, sorry. I would report it to support. Maybe it will trigger something on their end.

    0
  • Jerry Hall

    I did report it to support.  This afternoon I tried to break the system by rapid testing of duplicates, edits and moves using internal catalogued folders and external SSD

    Preference to create sidecars was set from ON1 when I did the initial install.  Used 3 fresh raw files (385, 386, 387) in working folders on uncatalogued SSD and catalogued internal drive.  Everything worked ok. .on1 files showed extensive text info presumably describing edits.

    My notes are here: https://docs.google.com/document/d/1Wc2RBtynbUz0QHG1-jRicOPQvkD0lxh4b2HB2QlbGmw/edit?usp=sharing

    Only odd thing was a delay of a few minutes for the edit text to show up in the external SSD .on1 file.  I later tried to exploit the delay by quick drag and drop to the internal catalogued folder but it worked fine as far as I could see.

    But I wonder if a timing issue could be related to some of the reported problems.

    I am going ahead with my plan above and un-catalog all folders.



    0
  • Brian Lawson Community moderator

    Your link above does not work.

    0
  • Jerry Hall

    Fixed.  Not very significant tho.

    0

Please sign in to leave a comment.