Scanning progress for databases

Introduction

This article serves to explain the logic behind how database scan progress is tracked, calculated, and updated by Enterprise Recon.

 

Calculation

The scanning engine calculates progress by the following order:

  1. Determine the total number of Tables to scan
  2. Assume every Table would be scanned twice (as scanning would be done twice on any Row that contains a BLOB). We will end up with 2x Table slices where BLOBs would be placed in the last half slice.
  3. Initialize progress = Zero-based index of the Table being scanned divided by total number of Tables
  4. Check if we're currently processing any BLOBs; if yes, that means we've finished scanning the 1st slice.
  5. In the case where we're scanning BLOBs, include the progress of the BLOB scan by the number of BLOBs - disregarding actual BLOB sizes.

Example

We have 5 Tables. ER is currently scanning 3rd Table and processing the 30th BLOB (our of 100) of that Table.

  1. Tables = 5
  2. Slices = 10
  3. Current Table = 3
    Progress = (Current Table - 1) ÷ Tables = 40%
  4. Progress = Progress + 1 ÷ Slices = 40% + 10% = 50%
  5. Progress = Progress + (30 ÷ 100 x 1 ÷ Slices) = 50 + 0.3 x 10 = 50% + 3% = 53%

 

FAQ

In the case where my database has Table A with 1,000,000 records and Table B with 1,000 records, will Enterprise Recon regard the completion of Table A as;
(a) 50% done?
(b) A more accurate progress of 90%?

The answer would be (a).
We've explicitly avoided implementing (b) for performance reasons.

Does ER update the current progress only after an entire Table/Schema/Catalog is scanned or does it update for each individual record scanned?

Progress is updated while processing BLOBs (if any) for the Table and/or when a Table finishes scanning.

Is it normal for my database scan progress to be stuck at a certain percentage for an unusually long period of time?

Yes, the larger your Tables the more you would experience this.
This is because during the first stage of processing/scanning a new Table, the scanning engine will be processing the whole Table as raw data and will not check/report back the progress to ER.
It will only check for progress whilst scanning BLOBs in the second stage of a Table scan.

 

All information in this article is accurate and true as of the last edited date.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.