0612: "Estimation"

[Zorlin]

Sweet! The return of excellent XKCD comics!

Yeah, that was a rough weekend.

I tip my metaphorical hat to you, good sir.

Apocalypto the Bunny approves of your comment too.

[philsov]

I just like the progress bars that go based off of number of files completed.

Sure, 99 of those files are a few kbs, they can be installed quickly.

But this last file... it's a gig or so. Enjoy the 99% for a while!

[FoolishOwl]

I've seen a lot of unstable time estimates in progress bars, but it's specifically the progress bar for Windows file copy operations that seems to have this sort of absurd fluctuation. I can understand fluctuating time estimates for downloading a file, in which the bandwidth may vary from moment to moment, there may be some compression that varies in effectiveness, etc. But all the information needed for a good estimate is available to the Windows file manager when copying files locally -- the actual file size, the speed of the drive, and so on -- so there's really no excuse for bad estimates.

It goes to show what happens when the source code is a secret.

[Mr. Burke]

The algorithm that Microsoft uses for time estimates must really be off. Might be time to think of a new one?

That would be “Calculate speed of previous data, extrapolate for remaining data.”
Hard to improve without clearvoyance. Harddisks are not as deterministic as you'd like them. Specifically, copying many small files probably takes longer than copying one big one.

[Clumpy]

This is the best XKCD in ages.

[capncanuck]

Gah, lost by a few seconds.

Anyway.

I generally experience this with torrents, but not much else.

Or you lost by a Plank time length.

[sircheeziusbreezius]

A more proper estimate would be a range, like "between 2 and 5 hours". The estimates should become more accurate over time, too.

God also shouldn't be crazy, provided I'm not crazy for maybe believing in a god.

[Ezbez]

My favorite is watching other people look at those progress bars. "Well now it says 5 minutes. It used to say 1 minute. The speed has been decreasing. I think it will be done in about 10 minutes."

Now if we just recurse and have someone watching them, making estimates based off their estimates, and someone watching him, etc.... we'd have either a supremely accurate or terribly inaccurate estimate. I'd guess that it would be the latter.

[Bright Shadows]

This comic is supremely awesome.

Y/N

[scikidus]

The number of applications which simply divide total file size by download speed to determine duration is shocking. Are rolling averages so hard?

Then again, are they that much more effective?

[10nitro]

Gah, lost by a few seconds.

Anyway.

I generally experience this with torrents, but not much else.

Or you lost by a Plank time length.

or you lost twenty dollars and your self respect. I did.

[Faranya]

I don't pay much attention to the time estimates. I just want a loading bar that accuratly reflects the current progress attained in relation to the total progress necessary for the job to be done.

I have had it up to here (image my hand is slightly above my eyebrows) with "loading" bars that simply cycle a small graphic through them over and over again while giving no real indication of the amount of work done. Seriously, can't the energy being used to show me a completely useless animated bar be instead use to get the job done faster?

[arachnophilia]

On my old Win95 machine with dial up I had made the mistake of trying to download a file that was far too large for my machine.... ...It MAY have told me that I had 399 years remaining <_<... Worst. Download. Ever.

that sounds about right for dialup.

[Radical Pi]

lol this is so true for me, and such perfect timing for this comic.

Today I was clearing off a hard drive for an old Win98 computer to dual-boot a lightweight linux, and the time remaining number kept climbing up higher and higher. I got a cell phone picture (horrible quality) at 2.5 million MINUTES. It peaked around 5 million minutes, then instantly dropped back down to 5 seconds remaining I think Microsoft was getting back at us for making fun of their estimation techniques

[10nitro]

I don't pay much attention to the time estimates. I just want a loading bar that accuratly reflects the current progress attained in relation to the total progress necessary for the job to be done.

I have had it up to here (image my hand is slightly above my eyebrows) with "loading" bars that simply cycle a small graphic through them over and over again while giving no real indication of the amount of work done. Seriously, can't the energy being used to show me a completely useless animated bar be instead use to get the job done faster?

The reason they show you an animation is to assure it hasn't frozen up. If the animation stops you have issues. This is why many programs have the animation side-by-side with the bar, with a slow-moving progress bar, you can't be sure if it's frozen or just slow. In Firefox the little circle of dots thing on the tab. In IE the earth/e in the upper right corner. In GNOME, the moving stripes on the load bar (with some themes, anyway). In the WinXP installer the little boxes in the bottom right. [edit]These boxes are sometimes useful, but sometimes they keep moving, and the mouse responding, even when installation freezes (it was stuck at "Setup will complete in approximately: 13 minutes / Installing devises" for >6 hours)[/edit]

[owenja]

Computationally estimating the time it takes to do something is surprisingly hard, and predictions that are both accurate and precise are literally impossible to make. As pointed out here, it's an instance of the halting problem.

Which is not to say that a decent progress bar is impossible, but as a rule it's a relatively small part of a program that would need a disproportionally large amount of effort to get right. Usually that time can be better spent on a different aspect of the program.

[anrieff]

Today I was clearing off a hard drive for an old Win98 computer to dual-boot a lightweight linux, and the time remaining number kept climbing up higher and higher. I got a cell phone picture (horrible quality) at 2.5 million MINUTES. It peaked around 5 million minutes, then instantly dropped back down to 5 seconds remaining I think Microsoft was getting back at us for making fun of their estimation techniques

My guess: what you hit is a bug in Win98, which uses 32-bit integers to calculate file sizes (or at least, when concerned with remaining time estimation). This hit me one day when I tried to move a handful of movies between partitions on a Windows 98 SE. The progress bar was fluctuating wildly, and the actual estimation more often negative than positive, with astronomic numbers such as 3501 years appearing intermittently.

[roderik]

seems like I've had it worse..
and I wasn't even looking at the time remaining.
the percentages it showed simply moved from 0% to 90% like they should, until they passed the 100% mark and went on to 300% then some time later dropped to -600% before finally finishing at around 900%.
it was horrible.

[TheGuyWithTheHat]

Hilarious!

[KiaserZohsay]

x = bytes copied so far
y = number of files copied so far
z = seconds elapsed

Data points are taken periodically during the process of copying multiple files (typically a few hundred).

Sx = Sum of all x data points
Sxx = Sum of the sqaures of all x data points
Etc.

Code: Select all

      Sx = Sx + CurrCompletedSize;
      Sxx = Sxx + 1.0 * CurrCompletedSize * CurrCompletedSize;
      Sxy = Sxy + 1.0 * CurrCompletedSize * CompletedCount;
      Sxz = Sxz + 1.0 * CurrCompletedSize * CompletedTime;
      Sy = Sy + CompletedCount;
      Syy = Syy + 1.0 * CompletedCount * CompletedCount;
      Syz = Syz + 1.0 * CompletedCount * CompletedTime;
      Sz = Sz + CompletedTime;
      N = N + 1.0;
      Det = Sxx*Syy*N - Sxx*Sy*Sy - Sxy*Sxy*N + Sxy*Sx*Sy + Sx*Sxy*Sy - Sx*Sx*Syy;
      if (Det > 0) then
        {
          A = (Sxz*Syy*N - Sxz*Sy*Sy + Syz*Sx*Sy - Syz*Sxy*N + Sz*Sxy*Sy - Sz*Sx*Syy) / Det;
          B = (Sxz*Sy*Sx - Sxz*Sxy*N + Syz*Sxx*N - Syz*Sx*Sx + Sz*Sx*Sxy -Sz*Sxx*Sy) / Det;
          C = (Sxz*Sxy*Sy - Sxz*Syy*Sx + Syz*Sxy*Sx - Syz*Sxx*Sy + Sz*Sxx*Syy - Sz*Sxy*Sxy) / Det;
          ProjectedTime = round( A*TotalSize/1024 + B*TotalCount + C );
        }


All of of this calculates constants A, B and C which are used to calculate projected total time for completion, which is used with elapsed time to estimate time remaining. On a typical copy run, the estimate fluctuates a little at the beginning, but then settles outs and counts down like a clock. My co-workers don't know what this code does, and my users don't even know it's there, but when my file copy dialog says you have 10 minutes left on 200 files, you better believe it will be done in ten minutes.

kz

[10nitro]

Computationally estimating the time it takes to do something is surprisingly hard, and predictions that are both accurate and precise are literally impossible to make. As pointed out here, it's an instance of the halting problem.

To a mathematician this is relevant, but not so much in practical use. If a progress bar is inaccurate... someone fails (bytescopied/totalbytes... not hard). From there there are 2 basic ways of doing the estimate, looking at the current speed and what's left to do, or a rolling average.

Even without spending a lot of time on it, this will be way more accurate than Windows (pseudo code)(progress returns a % in 0-1):

Code: Select all

i0=timeSinceStart(bar)/progress(bar)
while ( progress(bar) != 1 ) {
  i1=i0;
  i0=timeSinceStart(bar)/progress(bar)
  estimate=((i0+i1)/2)-timeSinceStart(bar)
}


although the guy above me's code is probably better. My point is making a decent estimate isn't hard.

[Ezbez]

10nitro: He stated that calculating time left not calculating amount done. Amount complete is usually quite accurate. (Though there are some counter examples, like Gametap which often tells me I am 101% of the way done and am downloading something like 7 billion gigabytes.)

[10nitro]

10nitro: He stated that calculating time left not calculating amount done. Amount complete is usually quite accurate. (Though there are some counter examples, like Gametap which often tells me I am 101% of the way done and am downloading something like 7 billion gigabytes.)

...I know the code I gave estimates the time left... I'm counting on amount complete to be accurate.
In my code progress(bar) returns the percentage complete (0.00-1.00).

[Wummi]

30 seconds, actually.

and i just installed the copy app from page 1, and it is in fact awsome²

also, its 64bit compatible... wooohooo

[nevetS]

First!

[moimoimoi]

As much as I'd love to bash Windows, it's not just them...

This is a well-known problem with Stuffit, not with Mac.

Computationally estimating the time it takes to do something is surprisingly hard, and predictions that are both accurate and precise are literally impossible to make.

That's why I love Mac's imprecise but accurate progress indicators. Its estimates are vague, but correct to the unit of time: e.g. "About 30 seconds".

[Tweedledum]

Good comic. In almost every way.

Perhaps it has much to do with hardware. Despite estimates of 14+ hours in my copying time, It never took any of my computers over 9 minutes to do anything less than 14GB.

*Shrugs*

[Kimmeh]

Also I'm another who is highly against putting the punctuation inside the quotes. It makes no sense! There was no punctuation there in the original quote! D:

I agree. There should only be punctuation in the quote if the punctuation is part of the quote. For example if you quote a full sentence.

But what do you do if the quote is both a full sentence (so it should end with a dot) and the end of the sentence (so there should be a dot behind the quote). Adding both looks aweful.

Alice said: "Let's go to the beach.".

But leaving one out feels wrong, to me. They are both full sentences, and should both end with a dot. In my native language (Dutch) the rule is the leave out the first dot in this case. And that is probably the best solution. But sometimes leaving both might not look bad, so what to do then?

Alice said: "Let's go to the beach!", but I did not feel like going.

This quote is problematic.

Normally you'd put the period inside the quotes. Alice said, "Let's go to the beach."

[TheGuyWithTheHat]

Also I'm another who is highly against putting the punctuation inside the quotes. It makes no sense! There was no punctuation there in the original quote! D:

I agree. There should only be punctuation in the quote if the punctuation is part of the quote. For example if you quote a full sentence.

But what do you do if the quote is both a full sentence (so it should end with a dot) and the end of the sentence (so there should be a dot behind the quote). Adding both looks aweful.

Alice said: "Let's go to the beach.".

But leaving one out feels wrong, to me. They are both full sentences, and should both end with a dot. In my native language (Dutch) the rule is the leave out the first dot in this case. And that is probably the best solution. But sometimes leaving both might not look bad, so what to do then?

Alice said: "Let's go to the beach!", but I did not feel like going.

This quote is problematic.

Normally you'd put the period inside the quotes. Alice said, "Let's go to the beach."

What I do is take the . out of the quote and put it outside, like "I ran away from a raptor".

When people put quotes, they don't usually put the whole sentence, so the way I do it seems logical to me.

[gz3zbz]

This comic (0612: Estimation) scored a comment on the Building Windows 8 blog at MSDN;

We’re anticipating that many of you are going to want to know what we’ve done to improve the accuracy of the estimated time remaining for a copy to complete. (This has been the source of some pretty funny jokes over the years).

http://blogs.msdn.com/b/b8/archive/2011/08/23/improving-our-file-management-basics-copy-move-rename-and-delete.aspx