The Truth About Quickbooks Desktop Files – revised!
Whether you’ve been using QuickBooks™ for years or just a just a short time you probably noticed all the updates and versions Intuit releases. With all of that activity and attention you trust that your data is in good shape. After all, people have been relying on QuickBooks since 1992.
Well, it’s time to tell the truth about the health of your QuickBooks company file.
In a perfect world, your software and data would work smoothly, doing precisely what they are designed to do. Unfortunately, we live in a dynamic world–both natural and digital. Things degrade, no matter how cautious you think you are. It’s called entropy.
Simply defined, entropy is a gradual decline into disorder
The main factor that affects your file is insufficient RAM. QB needs MUCH MORE than you might realize. Intuit suggests a minimum of 8 GB for Enterprise. More will be better – the Verify and Rebuild processes cannot function properly or completely without an abundance of available RAM. William “Murph” Murphy, one of the few QB database gurus around, reminded me it means 8 GB RAM available to the QB application. If you can’t afford to purchase (and maintain!) this kind of server, you could consider some of the hosted platforms…I only have one to recommend. We’ll get there later in this post.
There are several symptoms of potential QuickBooks performance or corruption issues:
- File Size
- The number of DB File Fragments
- The earliest of the Versions Used on File
- Slow performance or responsiveness
But do note – I said symptoms, not causes. There are many in the QuickBooks world, including Intuit, who claim your file has problems because of its size. Not true.
Curious about your own file stats? Open your file and hit F2 on your keyboard. You’ll see the Product Information window. (In Figure 1, I’ve highlighted the items I’m referring to.)
Figure 1. Open QuickBooks and hit F2 on your keyboard to see the Product Info screen
Let’s review these factors, and the actions you can take to protect your data against corruption and loss.
There Is Such A Thing As Too Big
From our years of hands-on experience working with hundreds of QuickBooks files, we have established real-world limits on the sizes of QB files before corruption and performance become serious issues:
Pro: 250 MB
Premier: 350 MB
Enterprise: 1.25 GB
Intuit, the maker of QuickBooks, seems to have a different take on file size. For instance, some support agents have told our customers that Enterprise can be 4GB or larger. Perhaps in a sterile, carefully tended environment. But that’s not the way small businesses run. It’s the Wild West out there! We currently have a Pro customer whose file is almost 700 MB. It’s incredibly slow to open and to load into the data warehouse we use. For two years I’ve recommended that they start a new file. Now they are asking if integrated applications can sync more quickly. We design miracle reports, but we can’t help with that miracle!
If your company file is close to, or has exceeded, the above file size for your version, there are three options for you to take:
- File Split. If your file is otherwise healthy, we can refer you to Advanced Certified QuickBooks ProAdvisors who specialize in eliminating old data. It’s often faster and less expensive than creating a new file. This is probably not an option, though, if you use Advanced Inventory or Pricing Levels.
- New File Creation. If your file has corruptions, is older than 10 years, or a File Split will only create a temporary reduction, you will want a new file created. I cannot stress enough how critical it is to hire an experienced ProAdvisor to perform this project, as it’s an involved project. I have a colleague who wrote a 57-page document explaining most of the process. It really needs to be a coordinated effort.
- Hosting. But not just any hosting. You need a managed services platform that is designed explicitly to fix and then prevent file corruptions. We found one, and only one.
- Do Nothing. If your file is relatively healthy, you can temporarily do nothing. This is temporary, however, and you risk the loss of some or all of your data, even if you do regular backups. Besides, how do you really know if your file is truly healthy?
As a fellow business owner, I urge you to take this caveat very seriously. I have had a customer call me in a panic after receiving a scary error message from QuickBooks. The file was kaput and the only option was to start from scratch and re-enter months’ worth of transactions. (Though the hosting service I mention above can truly resolve this kind of crisis with their Advanced Recovery service!)
Issues Beyond File Size
For most businesses, file size isn’t the most common problem. As I mentioned, it’s actually just a symptom. The problem is simple: you’re using it. Opening and closing a file can lead to degradation and entropy. Software glitches, power surges, and just sharing or copying a file to a new location can cause corruption.
Let’s say you’ve been using QuickBooks for quite some time. In 10 years, having just 3 users who are opening and closing a file twice per day (because, you know… it crashes sometimes) means there were over 15,000 opportunities for something to go wrong.
If your file is larger than the sizes already mentioned, your file is at greater risk because it runs more slowly. More time opening, saving, and closing means more exposure to these risks. If we include bad user habits, such as not closing QuickBooks at the end of the day, leaving transactions open and incomplete, we see even more corruptions and broken links.
One indicator for this kind of issue is the number of DB File Fragments. See Graphic 1. File Fragments are when QuickBooks has started saving packets of data in non-adjacent space on the hard drive. The more spread out the related packets are, the slower the performance of QuickBooks. There is a simple way to reduce or eliminate the file fragments – see the section at the end of this article.
As you use QuickBooks on a day-to-day basis, you likely will not notice there’s a problem. Sure, it may glitch or crash now and then, but it starts right back up and your transactions are still there. But entropy is real, folks, and without the implementation of solid data policies, your data could be at risk. Fortunately, you can take steps to protect your data.
Implement Data Policies
Investigate the topic of data policies. Plenty of accounting systems don’t allow you to keep more than two or 3 years of historical data in the active system. Before QuickBooks came along, every accounting solution gave you two years of data MAX, and you had no choice – and that included Peachtree and Great Plains. You can always keep your old file for reference purposes. Something worth considering. I promise you – you do not need 25 years of accounting data in your active file.
If you are doing web sales, you do not need to keep 51,000 customer names in QuickBooks. They should be in a CRM while putting those names or some type of identifier into a custom field
I just have to share something I saw recently – one of our customer’s files has been in use since it was a Quicken file. And it’s 1.3 GB. You can bet that the owner of the company is going to get a strongly worded invitation in the very near future to get a new file created.
Early Detection and Correction
QuickBooks does have data verification and rebuild tools built in. These tools are easy to use. You can find them in the File | Utilities menu.
The problem is that oftentimes the software will report that “QuickBooks detected no problems with your data” and then you might believe it. Some customers run a “Verification” thinking that will do something to the file – but it doesn’t, and it doesn’t always report problems when they DO exist.
All of that makes the job of a trusted advisor much more difficult. Some customers don’t take our warnings seriously, or they ignore them altogether.
Age Is Just A Number… Or Is It?
In 2006, Intuit made a significant change to the data structure and the way it is accessed, while still accommodating the way existing customers had been using the software. This means there may be data in fields where it no longer belongs because it’s a different data type. For example, there may be text in a field designed for phone numbers. If your file has been in use since 2006 or earlier, your data could be at risk. In this case we recommend a New File Creation. One customer with a 2 GB file decided to cancel a report project he’d already paid for instead of cleaning up the data that prevented the file from synchronizing with an integrated application. No joke.
Why File Care is Important
There are many reasons file care is important – but the most dangerous is that you can’t tell it is corrupted.
In 2013, I discovered a tool called QQube that makes it easier to access the data in QuickBooks. It uses the software development kit (SDK) and the Custom Reporting function to extract data from the system. The SDK sniffs out data corruption, never lies about it, and simply won’t transfer the data. Initially, I was frustrated by this. But after finding so many files that had otherwise undetectable issues, I realized it’s the only early warning system we’ve got.
Good User Habits Can Help
Your QuickBooks users can help with file care every day. This means always closing all open windows before exiting the program entirely. Many users who access QuickBooks via Remote Desktop (formerly known as Terminal Services) only close the Remote Desktop window – and forget to exit out of QuickBooks. This can cause issues.
Another cause of corrupted files is weak or faulty network connections – which can be caused by a loose or broken Ethernet cable or intermittent WiFi connections. These dodgy connections interrupt the process of saving transactions.
The QuickBooks Backup Process – Use It
There is something very useful accomplished by the QuickBooks built-in backup process. It takes changes in the TLG file and writes them to the database. This will help with corrupted memorized reports, and other important functions.
Updates – Timing is Everything
You may have noticed that QuickBooks sends out frequent fixes and updates. With every release, there are fixes and changes under the hood. When you do not establish a strict software update policy and process, QuickBooks allows users to continue using older releases. This, too, can cause problems.
The Real Cost Of QuickBooks
Selecting an accounting system to suit your operational needs – and your budget – is obviously critical. When you start out small, QuickBooks can usually handle the requirements. The problems start once significant growth begins – or when no one is tending to file care.
It’s the choice between pay now – or pay A LOT later. In our view, if you opt to use QuickBooks, you’re choosing to pay later. What you’ll pay for is a new file creation or file reduction every few years, to the tune of $2,000 to $20,000, depending on how much historical data you truly need to keep in your production file.
QuickBooks Desktop is an attractive, affordable option. But you need to understand how to care for your valuable data. It simply is not “set it and forget it.” Here are tips you can implement to help protect your business:
- Use Solid State Hard Drives – and ensure there is twice as much RAM as Intuit suggests
- Set and follow a regular Rebuild schedule
- Read here for excellent instructions and best practices for a Rebuild
- Understand that running a “Verify” doesn’t do anything to the file – and may not detect significant issues
- Perform a “Portable Copy Round Trip” to reduce the DB File Fragments to under 10 (but don’t bother running Defrag, especially on a Solid State Drive)
- Run the Backup process every night – it “writes” the TLG file to the database
- Turn Off the Audit Trail
- Don’t use the Condense feature – except, perhaps, on the Audit Trail if you’re using it
- Delay upgrading to the newest year of QuickBooks until around 6 months after its release
- Create and adhere to an update policy that:
- Disallows automatic updates
- Only allows IT to perform the update
- Updates everyone on the next login
- Don’t assume your IT provider knows how to manage and maintain QuickBooks
- Respect the minimum hardware specifications, especially for RAM, processing speed, and disk size
- Use a Solid State Hard Drive
- If your file is larger than recommended in this article or older than 10 years, bite the bullet, engage a pro, and get a fresh new file created
The Ultimate File Protection
I found a miracle that will help you protect your file way beyond a 1.25 GB size limit. Last year I weighed in on a file size debate on Hector Garcia’s Facebook group. Someone countered with “I can run files of 10GB or greater as fast as lightning.” That’s quite a claim! I quickly got in touch to learn more. Sure enough, this enterprising individual has built a hosting platform to handle large files AND eliminate and prevent corruption. After all, QuickBooks uses Sybase SQL Anywhere. That is a world-class database management system and can handle TERABYTES of data.
He also built an automated Advanced Recovery tool. So if your file won’t open, or there are serious problems, he can fix them. And he is, as far as I have discovered, the only person remaining who can and will fix unrecoverable files. However, the root cause of corruption, 9 times out of 10, is insufficient resources on the host machines. So if he fixes your file and you put it back on your existing system, it’s only a matter of time before issues reappear.
I should say that the other primary cause of these kinds of problems is Intuit’s insufficient error handling in the programming. But we can’t fix that!
If QuickBooks crashes while saving a transaction, that will most likely corrupt something. There’s a bit of a ripple effect too. When a crash causes corruption, the next time the program processes over that corrupted data another crash can occur and cause more damage.
If you want to know more, please drop me an email or leave a comment below – I’d be happy to make an introduction, and it helps to have an initial conversation about your specific situation.
We Love QuickBooks, But…
It seems like Intuit doesn’t instruct end users how to care for their files until there is a problem. And very often, that’s too late and the problem cannot be resolved. While I appreciate that we live in a dynamic world, I consider this to be unethical. A few years ago Intuit ran a QuickBooks campaign with the slogan, “Own It.” In my opinion, if you’re going to own QB, you’d better be willing to own the limitations that accompany the very reasonable pricing. It’s my hope that Intuit owns it, too, by:
- Telling customers the truth about file limitations
- Instructing customers how to care for their QuickBooks files
- Removing the condense feature
- Disallowing the use of multiple releases on the same file and network
Most business owners, and even many Certified QuickBooks ProAdvisors, are genuinely surprised to learn there’s a problem with the files. After all, it looks fine, right? But the truth is it just may not be. The older your file, the longer you use it, and the increased entropy that results may be putting you on the brink of a major league data crisis.
Contact us for a list of amazing ProAdvisors who specialize in New File Creation and File Splits!
Already a Datisfy Customer? Learn about our Report & Support Plans so we can help you with regular file care!